Image processing apparatus, image processing method, and program

ABSTRACT

An information processing system that controls a display to display an image generated based on a manipulation target image and an image based on a manipulation target region that is set based on a detection of an object included in acquired image data.

This application is a continuation of U.S. application Ser. No.15/337,029, filed Oct. 28, 2016, which is a continuation of U.S.application Ser. No. 14/380,812, filed Aug. 25, 2014, which is anational stage of International Application No. PCT/JP13/03116, filedMay 16, 2013, and claims the benefit of priority from Japanese PatentApplication No. 2012-133786, filed Jun. 13, 2012, the entire contents ofwhich each are incorporated herein by reference.

TECHNOLOGY FIELD

The present disclosure relates to an image processing apparatus, animage processing method, and a program.

BACKGROUND ART

In recent years, higher functionality or multi-functionality of devicessuch as electronic devices is progressing more and more, and content ofinstructions that has to be given to a device for a user to use thedevice is more diversified and complicated. Under these circumstances,in many devices, a GUI (Graphical User Interface) is adopted as a userinterface, and enables a user to give instructions to the device moreeasily than when using a device with a CUI (Character User Interface).However, for example, for users who are not familiar with a manipulationusing a pointing device such as a mouse, the GUI whose manipulation isperformed using the pointing device is not necessarily a user interfacewith excellent maneuverability.

Under these circumstances, technology related to a user interface forimproving maneuverability has been developed. An example of technologyfor displaying, on a display screen, an image in which a captured imageand an object image related to an object associated with a predeterminedevent are combined may include technology described in PTL 1.

CITATION LIST Patent Literature PTL 1: Japanese Patent ApplicationLaid-Open No. 2005-216061 SUMMARY Technical Problem

An image in which a captured image (a mirror image based on a capturedimage; the same applies hereinafter) and an object image related to anobject associated with a predetermined event are combined is displayedon a display screen, for example, as in a case in which the technologydescribed in PTL 1 is used, making it possible to feed a gestureoperation of a user back to the user.

However, for example, the technology described in PTL 1 is intended tochange a position in which the object image is combined according to aposition of the user detected from the captured image. Accordingly, forexample, when a manipulation target image including an object that is amanipulation target that can be manipulated by the user (hereinafterreferred to as “manipulation target object”), such as a button or anicon, like a web page and the captured image are combined, it isdifficult to apply, for example, the technology described in PTL 1.Thus, for example, a process related to the technology described in PTL1 is not necessarily a gesture recognition-conscious process.

In the present disclosure, a new and improved image processingapparatus, image processing method, and program capable of displaying acaptured image and a manipulation target image on a display screen whilepreventing degradation of maneuverability of a user are proposed.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan information processing system including: circuitry configured to:control a display to display an image generated based on a manipulationtarget image and an image based on a manipulation target region that isset based on a detection of an object included in acquired image data.

Further, according to another embodiment of the present disclosure,there is provided a method performed by an information processingsystem, the method including: controlling a display to display an imagegenerated based on a manipulation target image and an image based on amanipulation target region that is set based on a detection of an objectincluded in acquired image data.

Further, according to the embodiment of the present disclosure, there isprovided a non-transitory computer-readable medium includingcomputer-program instructions, which when executed by an informationprocessing system, cause the information processing system to: control adisplay to display an image generated based on a manipulation targetimage and an image based on a manipulation target region that is setbased on a detection of an object included in acquired image data.

Advantageous Effects of Invention

According to the embodiment of the present disclosure, it is possible todisplay the captured image and the manipulation target image on thedisplay screen while preventing degradation of the maneuverability ofthe user.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an illustrative diagram illustrating an example of a regionsetting process according to the present embodiment.

FIG. 2 is an illustrative diagram illustrating an example of a regionsetting process according to the present embodiment.

FIG. 3 is an illustrative diagram illustrating an example of a regionsetting process according to the present embodiment.

FIG. 4 is an illustrative diagram illustrating an example of a regionsetting process according to the present embodiment.

FIG. 5 is an illustrative diagram illustrating an example of amanipulation target region set by an image processing apparatusaccording to the present embodiment.

FIG. 6 is an illustrative diagram illustrating an example of an effectof an image processing apparatus according to the present embodimentseparately setting a display target region and a manipulation targetregion.

FIG. 7 is an illustrative diagram illustrating another example of theregion setting process according to the present embodiment.

FIG. 8 is an illustrative diagram illustrating an example of a method ofnotifying a user that the user is closer to the outside of a range ofrecognition in a manipulation target region according to the presentembodiment.

FIG. 9A is an illustrative diagram illustrating an example of a combinedimage to be displayed on a display screen by the image processingapparatus according to the present embodiment.

FIG. 9B is an illustrative diagram illustrating an example of a combinedimage to be displayed on a display screen by the image processingapparatus according to the present embodiment.

FIG. 10A is an illustrative diagram illustrating an example of acombined image to be displayed on a display screen by the imageprocessing apparatus according to the present embodiment.

FIG. 10B is an illustrative diagram illustrating an example of acombined image to be displayed on a display screen by the imageprocessing apparatus according to the present embodiment.

FIG. 11 is an illustrative diagram illustrating an example of a combinedimage to be displayed on a display screen by the image processingapparatus according to the present embodiment.

FIG. 12 is an illustrative diagram illustrating an example of a combinedimage to be displayed on a display screen by the image processingapparatus according to the present embodiment.

FIG. 13 is an illustrative diagram illustrating an example of amanipulation target image according to the present embodiment.

FIG. 14 is an illustrative diagram illustrating another example of imageprocessing in the image processing apparatus according to the presentembodiment.

FIG. 15 is an illustrative diagram illustrating an example of apredetermined region larger than a display target region according tothe present embodiment.

FIG. 16 is an illustrative diagram illustrating another example of imageprocessing in the image processing apparatus according to the presentembodiment.

FIG. 17 is an illustrative diagram illustrating another example of imageprocessing in the image processing apparatus according to the presentembodiment.

FIG. 18 is an illustrative diagram illustrating another example of imageprocessing in the image processing apparatus according to the presentembodiment.

FIG. 19 is an illustrative diagram illustrating an example of anotification process in the image processing apparatus according to thepresent embodiment.

FIG. 20 is an illustrative diagram illustrating an example of anotification process in the image processing apparatus according to thepresent embodiment.

FIG. 21 is an illustrative diagram illustrating an example of anotification process in the image processing apparatus according to thepresent embodiment.

FIG. 22 is a flow diagram illustrating an example of a process accordingto an image processing method according to the present embodiment.

FIG. 23 is a block diagram illustrating an example of a configuration ofthe image processing apparatus according to the present embodiment.

FIG. 24 is an illustrative diagram illustrating an example of a hardwareconfiguration of the image processing apparatus according to the presentembodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Further, a description will be given in the following order:

1. Image processing method according to the present embodiment

2. Image processing apparatus according to the present embodiment

3. Program according to the present embodiment

Image Processing Method According to the Present Embodiment

An image processing method according to the present embodiment will bedescribed prior to a description of a configuration of the imageprocessing apparatus according to the present embodiment. Further, inthe following description, it is assumed that the image processingapparatus according to the present embodiment performs a processaccording to the image processing method according to the presentembodiment.

[1] Overview of Image Processing Method According to the PresentEmbodiment

As a method of displaying a captured image for feeding a gestureoperation back to a user and a manipulation target image on a displayscreen, for example, a method for combining the captured image with themanipulation target image and displaying a resultant image on thedisplay screen may be considered.

However, when the method of simply combining the captured image with themanipulation target image and displaying a resultant image on thedisplay screen is used, for example, it may be difficult for the user tomanipulate a manipulation target object arranged within the displayscreen. An example of a case in which it is difficult for the user tomanipulate the manipulation target object arranged within the displayscreen may include a case in which a hand of the user does not reach themanipulation target object arranged at a screen edge of the displayscreen even when the user moves the hand. Thus, when the method ofsimply combining the captured image with the manipulation target imageand displaying a resultant image on the display screen is used, themaneuverability of the user may be degraded.

Therefore, the image processing apparatus according to the presentembodiment sets a manipulation target region in a part of the capturedimage (a region setting process). Further, the image processingapparatus according to the present embodiment combines an image of aregion corresponding to the manipulation target region to be displayedon the display screen in the captured image (hereinafter referred to asa “display target region”) with the manipulation target image (imageprocessing). Also, the image processing apparatus according to thepresent embodiment displays a combined image on the display screen (adisplay control process).

Here, the captured image according to the present embodiment is, forexample, an image obtained by performing mirror image processing (ahorizontal inversion process) on an image captured by an externalimaging apparatus or by an imaging unit (which will be described below)when the image processing apparatus according to the present embodimentincludes the imaging unit (which will be described below). The capturedimage according to the present embodiment serves to feed a gestureoperation back to a user. As the image with which the captured image hasbeen combined is displayed on the display screen, a user can visuallyconfirm, for example, content of his or her gesture, such as performinga manipulation of sliding his or her hand or pressing using the hand orperforming an enlarging manipulation, a reducing manipulation or amoving manipulation using multiple fingers. Here, the mirror imageprocessing may be performed by the image processing apparatus accordingto the present embodiment or may be performed by the external apparatussuch as an imaging apparatus that has performed imaging. Further, anexample of the captured image according to the present embodiment mayinclude a moving image (or a plurality of still images). Hereinafter, acase in which the captured image according to the present embodiment isa moving image including a plurality of frame images will be describedby way of example.

An example of the external imaging apparatus and the imaging unit (whichwill be described below) according to the present embodiment may includea camera capable of acquiring RGB luminance information, but theexternal imaging apparatus and the imaging unit (which will be describedbelow) according to the present embodiment are not limited thereto. Forexample, the external imaging apparatus and the imaging unit (which willbe described below) according to the present embodiment can be a specialcamera capable of acquiring depth information, such as a depth camera.

Further, the captured image according to the present embodiment is notlimited to the image subjected to the mirror image processing(horizontal inversion processing). For example, if the mirror image iscaptured in the imaging apparatus or the imaging unit (which will bedescribed below), the captured image according to the present embodimentmay be the captured image itself.

Further, the manipulation target region according to the presentembodiment is a region in the captured image set in a part of thecaptured image, and is a region in which gesture recognition isperformed. Here, a process related to the gesture recognition may beperformed by the image processing apparatus according to the presentembodiment or may be performed by an apparatus external to the imageprocessing apparatus according to the present embodiment.

Further, the manipulation target image according to the presentembodiment, for example, is an image including a manipulation targetobject manipulated according to a result of gesture recognitionaccording to a gesture manipulation of the user. Here, an example of themanipulation target image according to the present embodiment mayinclude an image showing a web page or a menu screen and a manipulationscreen for each device. Further, examples of the manipulation targetobject according to the present embodiment may include a button, anicon, and a link such as a hyperlink. Further, as described above, theprocess related to the gesture recognition may be performed by the imageprocessing apparatus according to the present embodiment or may beperformed by an apparatus external to the image processing apparatusaccording to the present embodiment.

Further, an example of a UI (User Interface; icon size, layout or thelike) shown in the manipulation target image according to the presentembodiment may include the same UI (i.e., a fixed UI) in a case of amanipulation by a gesture or sound and a case of a manipulation by anexternal manipulation device such as a remote controller or amanipulation by a GUI, the UI shown in the manipulation target imageaccording to the present embodiment is not limited thereto. For example,in the manipulation target image according to the present embodiment,the UI may be different in the case of a manipulation by a gesture orsound and the case of a manipulation by an external manipulation devicesuch as a remote controller. An example of a case in which the UI isdifferent may include a case in which 4*4 icons are displayed in thecase of a manipulation by an external manipulation device such as aremote controller and 2*2 or 3*3 icons are displayed for easiermanipulation in the case of a manipulation by a gesture or sound.Further, for example, in the case of the manipulation by a gesture, amanipulation target object for assisting in a gesture manipulation(e.g., a control panel) may be included in the manipulation target imageaccording to the present embodiment.

Further, the display target region according to the present embodimentis a region corresponding to the manipulation target region set in aregion setting process, which will be described below. An example of thedisplay target region according to the present embodiment may include aregion matching the manipulation target region, but the display targetregion according to the present embodiment is not limited thereto. Forexample, the display target region according to the present embodimentmay be a region smaller than the manipulation target region. An exampleof a relationship between the manipulation target region and the displaytarget region according to the present embodiment will be describedbelow.

Further, an example of the display screen according to the presentembodiment may include a display screen in an external display device ora display screen in a display unit (which will be described below) whenthe image processing apparatus according to the present embodimentincludes the display unit (which will be described below).

The image processing apparatus according to the present embodimentperforms, for example, (1) a region setting process, (2) imageprocessing, and (3) a display control process, as processes according tothe image processing method according to the present embodiment.Hereinafter, the processes according to the image processing methodaccording to the present embodiment will be described in greater detail.

(1) Region Setting Process

The image processing apparatus according to the present embodiment setsthe manipulation target region in a part of the captured image.

The image processing apparatus according to the present embodiment, forexample, determines whether a start condition of gesture recognition(trigger of gesture recognition start) is satisfied, and sets themanipulation target region if it is determined that the start conditionof gesture recognition is satisfied. Here, examples of the startcondition of gesture recognition according to the present embodiment mayinclude a predetermined gesture such as a hand gesture or hand holdingbeing recognized, predetermined sound such as “Hello TV” beingrecognized, and a manipulation signal based on a start manipulation forgesture recognition (an example of the user manipulation), deliveredfrom a manipulation unit (which will be described below) or an externalmanipulation device such as a remote controller, being detected. Theimage processing apparatus according to the present embodimentdetermines whether a predetermined gesture has been recognized, forexample, based on a signal indicating a gesture recognition result (whenthe image processing apparatus according to the present embodiment has agesture recognition function based on the captured image) or a gesturerecognition result acquired from an external apparatus. Further, theimage processing apparatus according to the present embodimentdetermines whether predetermined sound has been recognized, for example,based on a signal indicating a sound recognition result (when the imageprocessing apparatus according to the present embodiment has a soundrecognition function based on a sound signal) or a sound recognitionresult acquired from an external apparatus.

Further, if it is determined that a start condition of gesturerecognition corresponding to a plurality of users has been satisfied,the image processing apparatus according to the present embodiment mayset the manipulation target region, for example, according to a priorityset for each user. Here, examples of a method of setting the prioritymay include a variety of methods, such as a method of setting a highpriority for a first recognized user or a last recognized user or amethod of setting a priority based on a user manipulation.

Further, the region setting process in the image processing apparatusaccording to the present embodiment is not limited to setting themanipulation target region if it is determined that the start conditionof gesture recognition is satisfied. For example, the image processingapparatus according to the present embodiment may set a region that is acandidate for the manipulation target region regularly/irregularly. Ifthe region that is a candidate for the manipulation target region is setregularly/irregularly, the image processing apparatus according to thepresent embodiment sets, as the manipulation target region, the mostrecently set region that is the candidate for the manipulation targetregion, for example, when it is determined that the start condition ofgesture recognition is satisfied.

(1-1) First Example of Region Setting Process

The image processing apparatus according to the present embodiment sets,for example, a manipulation target region based on a position and a sizeof a detection target in a captured image.

FIG. 1 is an illustrative diagram illustrating an example of a regionsetting process according to the present embodiment. Here, FIG. 1illustrates an example of a case in which the image processing apparatusaccording to the present embodiment sets a manipulation target regionusing a hand of a user as a detection target. Further, A shown in FIG. 1illustrates an example of the captured image, and B shown in FIG. 1illustrates an example of the manipulation target region to be set.

The image processing apparatus according to the present embodiment, forexample, detects a location and a size of the hand from the capturedimage and sets the manipulation target region based on detection valuesof the detected position and size of the hand.

Here, the image processing apparatus according to the presentembodiment, for example, identifies a region including the hand (anexample of the detection target) based on a result of an edge detectionprocess for the captured image or a result of a process of detecting apredetermined color, such as a skin color, for the captured image, todetect the position and the size of the hand from the captured image.Further, the process of detecting the position and the size of the handin the image processing apparatus according to the present embodiment isnot limited to the above process. For example, the image processingapparatus according to the present embodiment can detect the positionand the size of the hand from the captured image through any processcapable of detecting the position and the size of the hand.

Further, the image processing apparatus according to the presentembodiment sets, for example, as the manipulation target region, aregion having any size including the detected hand, which is smallerthan the entire captured image.

Further, when the image processing apparatus according to the presentembodiment detects the position and the size of the hand from thecaptured image to set the manipulation target region, a manipulationtarget region expected by a manipulator (a user) may be set outside anangle of view in an external imaging apparatus or imaging unit (whichwill be described below). The image processing apparatus according tothe present embodiment determines whether the manipulation target regionset based on the position and the size of the hand detected from thecaptured image is set outside of the view angle. If the manipulationtarget region is determined to be set outside the view angle, the imageprocessing apparatus may set, for example, the manipulation targetregion in a position opposite in the horizontal direction to thedetected hand (an position opposite to a position of the user).

FIG. 2 is an illustrative diagram illustrating an example of the regionsetting process according to the present embodiment. Here, FIG. 2illustrates an example of the manipulation target region set when theimage processing apparatus according to the present embodimentdetermines whether the manipulation target region is outside the viewangle. Further, FIG. 2 illustrates an example of the captured image.

When the manipulation target region set based on the location and thesize of the hand detected from the captured image is set outside theangle of view, for example, as shown in A of FIG. 2, the imageprocessing apparatus according to the present embodiment sets themanipulation target region in a position opposite in the horizontaldirection to the detected hand, as shown in B of FIG. 2.

The image processing apparatus according to the present embodiment sets,for example, the manipulation target region based on the position andsize of the hand in the captured image, as described above. Further, thedetection target whose position and size are detected by the imageprocessing apparatus according to the present embodiment is not limitedto the hand of the user, for example, as shown in FIG. 1. For example,the image processing apparatus according to the present embodiment maydetect a position and size of a face of the user and set themanipulation target region based on detection values of the detectedposition and size of the face.

Here, the image processing apparatus according to the present embodimentdetects, for example, the position and size of the face from thecaptured image by performing any face detection process, such as adetection method based on brightness in the captured image.

Further, the image processing apparatus according to the presentembodiment, for example, sets a region smaller than the entire capturedimage as the manipulation target region based on the detection values ofthe detected position and size of the face. Here, the image processingapparatus according to the present embodiment sets, for example, as themanipulation target region, a region having any size including thedetected face, which is smaller than the entire captured image.

Further, the manipulation target region to be set when the imageprocessing apparatus according to the present embodiment detects theposition and size of the face is not limited to the above manipulationtarget region. For example, the image processing apparatus according tothe present embodiment may set, as the manipulation target region, aregion obtained by shifting, in a horizontal direction, a region havingany size including the detected face according to the handedness of theuser corresponding to the detected face. The image processing apparatusaccording to the present embodiment, for example, sets the manipulationtarget region in a position corresponding to the handedness of the usercorresponding to the detected face, based on handedness information(data) indicating the handedness of the user corresponding to thedetected face. Here, the handedness information according to the presentembodiment is stored for each user in a recording medium such as astorage unit (which will be described below), for example, based on aregistration manipulation of the user (an example of the usermanipulation). Further, the image processing apparatus according to thepresent embodiment reads the handedness information corresponding to theuser corresponding to the detected face, for example, from the storageunit (which will be described below) by identifying the usercorresponding to the face detected using face recognition technology.Also, the image processing apparatus according to the present embodimentsets the manipulation target region based on the read handednessinformation.

The image processing apparatus according to the present embodiment setsthe manipulation target region by detecting, as the detection target,one part of the user such as the hand or the face of the user, forexample, as described above. Further, the detection target whoseposition and size are detected by the image processing apparatusaccording to the present embodiment is not limited to one part of theuser, such as the hand or the face of the user. For example, the imageprocessing apparatus according to the present embodiment may set themanipulation target region by detecting a plurality of parts of the useras the detection targets.

For example, the image processing apparatus according to the presentembodiment detects the position and size of the hand of the user and theposition and the size of the face of the user, and sets the manipulationtarget region based on the detection values of the position and size ofthe detected hand and the detection values of the position and size ofthe detected face. FIG. 3 is an illustrative diagram illustrating anexample of the region setting process according to the presentembodiment. Here, FIG. 3 illustrates an example of a case in which theimage processing apparatus according to the present embodiment sets themanipulation target region using the hand of the user and the face ofthe user as detection targets. Further, FIG. 3 illustrates an example ofthe captured image.

When a captured image indicates that a user raises a right hand, forexample, as shown in FIG. 3, the image processing apparatus according tothe present embodiment sets the manipulation target region on a rightside of the detected face (a left side in a mirror image).

The image processing apparatus according to the present embodiment setsthe manipulation target region, for example, based on the position andsize of the detection target in the captured image, as described above.Here, as the image processing apparatus according to the presentembodiment uses, for example, the hand or the face of the user as thedetection target, a ratio of the set manipulation target region and thesize of the face or hand of the user included in the manipulation targetregion is kept constant. Thus, as the image processing apparatusaccording to the present embodiment uses, for example, the hand or theface of the user as the detection target, the user is able to perform amanipulation with the similar manipulation feeling, for example, evenwhen the user is in a position apart from an external imaging apparatusor imaging unit (which will be described below) or even when the user isin a position close to the external imaging apparatus or imaging unit(which will be described below).

(1-2) Second Example of Region Setting Process

The image processing apparatus according to the present embodiment setsthe manipulation target region, for example, based on posture of theuser detected from the captured image. The image processing apparatusaccording to the present embodiment obtains skeleton information (data)of the user using a process according to any posture estimationtechnology, such as estimating the posture of the user based onestimation of a part of the user using, for example, machine learning.Also, the image processing apparatus according to the present embodimentsets the manipulation target region by estimating an operating range ofthe user from the skeleton information.

Further, the process of setting the manipulation target region based onthe posture of the user in the image processing apparatus according tothe present embodiment is not limited to the above process. For example,the image processing apparatus according to the present embodiment mayfurther detect the position and size of the face of the user and set themanipulation target region based on the estimation result for theposture of the user and detection values of the detected position andsize of the face.

FIG. 4 is an illustrative diagram illustrating an example of the regionsetting process according to the present embodiment. Here, FIG. 4illustrates an example of a case in which the image processing apparatusaccording to the present embodiment sets the manipulation target regionbased on the estimation result for the posture of the user and thedetection values of the position and size of the detected face. Further,FIG. 4 illustrates an example of the captured image.

For example, if an extremely tilted face is detected when the userassumes a reclining posture as shown in FIG. 4, the image processingapparatus according to the present embodiment estimates a driving rangeof the hand of the user to set the manipulation target region. Forexample, in the example of FIG. 4, an example in which the manipulationtarget region is set in an upward direction of the detected face (upwardin a vertical direction), rather than the manipulation target regionbeing set in a position shifted in a horizontal direction from thedetected face, is illustrated.

Further, the image processing apparatus according to the presentembodiment may fix a manipulation target region that has been once set,or may dynamically change a manipulation target region according to amovement of the face, the hand or the like of the user to be detectedfrom the captured image.

(1-3) Third Example of Region Setting Process

The image processing apparatus according to the present embodiment setsthe manipulation target region, for example, based on the motion of thedetection target.

For example, the image processing apparatus according to the presentembodiment detects a predetermined gesture, such as a hand gesture suchas waving goodbye or drawing any shape in the air, and sets a regionincluding a movement region of a hand performing the gesture (an exampleof the detection target) as the manipulation target region. Further, thedetection target detected by the image processing apparatus according tothe present embodiment is not limited to a part of the user, such as thehand of the user. For example, the image processing apparatus accordingto the present embodiment may set, as the manipulation target region, aregion including a movement region of a predetermined device, such as arod-like device that the user has in his or her hand.

(1-4) Fourth Example of Region Setting Process

The image processing apparatus according to the present embodiment setsthe manipulation target region, for example, based on a shape of thedetection target and a size of the manipulation target object includedin a manipulation target image.

For example, in a case in which an object of a target on which gesturerecognition is performed (which may be hereinafter referred to as a“recognition target object”) is the hand of the user, when the size ofthe hand of the user included in the set manipulation target region ismuch larger than an icon included in the manipulation target image (anexample of the manipulation target object), it may be difficult for theuser to perform a desired manipulation using the hand. An example of acase in which it is difficult to perform a desired manipulation usingthe hand may include a case in which the user does not know where theuser is pointing when selecting content, for example, by a par. Further,the user does not know where the user is pointing in cases other thanthe case in which the user selects the content by a par, and, forexample, the user may not know where the user is pointing when the userselects content with a fingertip.

Therefore, the image processing apparatus according to the presentembodiment sets the manipulation target region, for example, so that adifference between the size of the detection target and the size of themanipulation target object is within a predetermined range, based on theshape of the detection target with which the user performs amanipulation and the size of the manipulation target object. Forexample, the image processing apparatus according to the presentembodiment sets the manipulation target region so that the size of thedetection target and the size of the manipulation target object areapproximately the same. Here, the predetermined range according to thepresent embodiment described above may be a fixed value that has beenset in advance or may a variable value that can be changed, for example,by the user.

(1-5) Fifth Example of Region Setting Process

For example, the image processing apparatus according to the presentembodiment may set the manipulation target region by combining aplurality of processes that can be combined among the first example ofthe region setting process to the fourth example of the region settingprocess, such as the first example of the region setting process and thesecond example of the region setting process, the first example of theregion setting process and the third example of the region settingprocess, and the first example of the region setting process and thefourth example of the region setting process.

(1-6) Sixth Example of Region Setting Process

The image processing apparatus according to the present embodiment, forexample, sets the manipulation target region based on one or two or moreof the position and size of the detection target in the captured image,the motion of the detection target, the posture of the user, the shapeof the detection target, and the size of the manipulation target objectincluded in the manipulation target image, as described above. Here, theimage processing apparatus according to the present embodiment, forexample, sets a region that matches a display target region according tothe present embodiment as the manipulation target region, but themanipulation target region set by the image processing apparatusaccording to the present embodiment is not limited to the region. Forexample, the image processing apparatus according to the presentembodiment may set, as the manipulation target region, a region smallerthan the entire captured image region and larger than the display targetregion.

As a specific example, the image processing apparatus according to thepresent embodiment calculates values related to the display targetregion using, for example, the following Math. 1 to Math. 4. Further,the following Math. 1 to Math. 4 show an example of calculation of thevalues related to the display target region when the display targetregion is a rectangular region. Here, “vw” shown in Math. 1 indicates awidth in a horizontal direction of the display target region, and “fw”shown in Math. 1 indicates a width (a known value) in a horizontaldirection of a rectangular region in which a face has been recognized inthe captured image. Further, “magnification ratio of visual feedback”shown in Math. 1 is a value to be set. Further, “vh” shown in Math. 2indicates a width in a vertical direction of the display target region,and “height ratio” shown in Math. 2 indicates a vertical width of thedisplay screen/a horizontal width of the display screen (a known value;for example, height ratio=9/16 in the case of a 16:9 screen). Further,“vx” shown in Math. 3 indicates an x coordinate of an upper left cornerof the display target region (a rectangular region), and “fcx” shown inMath. 3 indicates an x coordinate (a known value) of a center of therectangular region in which the face has been recognized in the capturedimage. Further, “offset value in x direction” shown in Math. 3 is avalue to be set. Further, “vy” shown in Math. 4 indicates a y coordinateof an upper left corner of the display target region (rectangularregion), and “fcy” shown in Math. 4 indicates a y coordinate (a knownvalue) of the center of the rectangular region in which the face hasbeen recognized in the captured image. Further, “fh” shown in Math. 4indicates a width (known value) in a vertical direction of therectangular region in which the face has been recognized in the capturedimage, and “offset value in y direction” shown in Math. 4 is a value tobe set. Further, “f” shown in Math. 3 and Math. 4 indicates a value tobe set (one of the values 0 to 9).

vw=fw*(magnification ratio of visual feedback)  (Math. 1)

vh=vw*height ratio  (Math. 2)

vx=fcx+fw*(offset value in x direction)−vw/2·f  (Math. 3)

vy=fcy+fh*(offset value in y direction)−vh/2·f  (Math. 4)

Further, the image processing apparatus according to the presentembodiment calculates values related to the manipulation target region,for example, using the following Math. 5 to Math. 8. Further, thefollowing Math. 5 to Math. 8 show an example of calculation of thevalues related to the manipulation target region when the manipulationtarget region is a rectangular region. Here, “rw” shown in Math. 5indicates a width in a horizontal direction of the manipulation targetregion, and “hw” shown in Math. 5 indicates a width (a known value) in ahorizontal direction of the rectangular region in which the hand hasbeen recognized in the captured image. Further, “scaling ratio of widthin horizontal direction” shown in Math. 5 is a parameter indicating asurplus with which a recognition range expands based on a size of thehand recognized in the captured image and is a value to be set. Further,“rh” shown in Math. 6 indicates a width in the vertical direction of themanipulation target region and “hh” shown in Math. 6 indicates a width(a known value) in the vertical direction of the rectangular region inwhich the hand has been recognized in the captured image. Further,“scaling ratio of width in vertical direction” shown in Math. 6indicates a parameter indicating a surplus with which a recognitionrange expands based on a size of the hand recognized in the capturedimage and is a value to be set. Further, “rx” shown in Math. 7 indicatesan x coordinate of an upper left corner of the manipulation targetregion (the rectangular region). Further, “ry” shown in Math. 8indicates a y coordinate of the upper left corner of the manipulationtarget region (the rectangular region).

rw=vw+hw*(scaling ratio of width in horizontal direction)  (Math. 5)

rh=vw+hh*(scaling ratio of width in vertical direction)  (Math. 6)

rx=vx+vw/2−rw/2  (Math. 7)

ry=vy+vh/2−rh/2  (Math. 8)

The image processing apparatus according to the present embodiment setsthe display target region, for example, by calculating the valuesrelated to the display target region using Math. 1 to Math. 4 describedabove. Further, the image processing apparatus according to the presentembodiment sets a region smaller than the region of the entire capturedimage and larger than the display target region as the manipulationtarget region, for example, by calculating the values related to themanipulation target region using Math. 5 to Math. 8 described above.Further, it is understood that the process related to the setting of themanipulation target region smaller than the region of the entirecaptured image and larger than the display target region according tothe present embodiment is not limited to the above process.

FIG. 5 is an illustrative diagram illustrating an example of themanipulation target region set by the image processing apparatusaccording to the present embodiment. Here, A shown in FIG. 5 indicates acaptured image. Further, B shown in FIG. 5 indicates an example of thedisplay target region according to the present embodiment, and C shownin FIG. 5 indicates an example of the manipulation target regionaccording to the present embodiment.

As shown in FIG. 5, the image processing apparatus according to thepresent embodiment sets a region smaller than the region of the entirecaptured image (A shown in FIG. 5) and larger than the display targetregion (B shown in FIG. 5) as the manipulation target region (C shown inFIG. 5). As the image processing apparatus according to the presentembodiment separately sets the display target region that is a regiondisplayed on the display screen and the manipulation target region thatis a region in which the gesture recognition is performed as shown inFIG. 5, for example, effects shown in (a) and (b) are obtained.

(a) Even when the user has conducted a gesture outside the displaytarget region displayed on the display screen, the apparatus thatperforms a process related to gesture recognition (the image processingapparatus according to the present embodiment and/or an externalapparatus; the same applies hereinafter) can recognize the gesture.

(b) Through (a), for example, the following manipulations using regionsof edges in the display target region displayed on the display screenare realized:

A selection manipulation to select manipulation target objects presentin the regions of edges at a top and bottom of the screen and at a leftand right of the screen

An auto scroll manipulation to cause a movement of a portion of an imagedisplayed on the display screen (e.g., only a specific manipulationtarget object) or transition to another image by moving the hand to thevicinity of the edge of the screen

A scroll manipulation to cause a movement of a portion of an imagedisplayed on the display screen (e.g., only a manipulation targetobject) or transition to another image through a gesture of striking(touching) the vicinity of the edge of the screen (an example of thegesture).

FIG. 6 is an illustrative diagram illustrating an example of an effectof the image processing apparatus according to the present embodimentseparately setting the display target region and the manipulation targetregion. Here, in FIG. 6, only the manipulation target image rather thanan image in which an image in the display target region and themanipulation target image in the captured image are combined by imageprocessing according to the present embodiment that will be describedbelow is illustrated for convenience of description. Further, AR1 shownin FIG. 6 indicates a region corresponding to the display target region.Further, AR2 shown in FIG. 6 indicates a region corresponding to apartial region in the manipulation target region, and a part of theregion AR2 is not included in the region AR1, as shown in FIG. 6.

For example, when the user (manipulator) moves the hand to the regionAR2 shown in FIG. 6 (an example of a region near the edge of thescreen), the apparatus that performs a process related to gesturerecognition, for example, recognizes that an auto scroll manipulationhas been performed. Also, if it is recognized that the auto scrollmanipulation has been performed, the apparatus that performs a processrelated to gesture recognition, for example, moves “TV,” “Web Browser,”“Media Player,” . . . , “Help Guide” that are manipulation targetobjects included in the manipulation target image illustrated in FIG. 6,in a direction M shown in FIG. 6 (an example of a case in which aportion of an image displayed on the display screen is moved). As themanipulation target objects included in the manipulation target imageshown in FIG. 6 are moved in the direction M shown in FIG. 6, forexample, other manipulation target objects are displayed on the displayscreen. Further, a change in the image displayed on the display screenwhen it is recognized that the auto-scroll manipulation has beenperformed is not limited to the example shown above. For example, if itis recognized that the auto scroll manipulation has been performed, theapparatus that performs a process related to gesture recognition maymove the entire manipulation target image shown in FIG. 6 in thedirection M shown in FIG. 6 and cause the image displayed on the displayscreen to transition from the manipulation target image shown in FIG. 6to the other manipulation target image (an example of a case in whichthe image displayed on the display screen transitions). As themanipulation target image shown in FIG. 6 is moved in the direction Mshown in FIG. 6 and caused to transition to the other manipulationtarget image, a manipulation target image different from themanipulation target image shown in FIG. 6 is displayed on the displayscreen.

Further, for example, if the user (manipulator) performs a gesture ofstriking (touching) the region AR2 shown in FIG. 6 (an example of theregion near the edge of the screen), the apparatus that performs aprocess related to gesture recognition recognizes, for example, that ascroll manipulation has been performed. Also, if it is recognized thatthe scroll manipulation has been performed, the apparatus that performsa process related to gesture recognition, for example, moves a specificmanipulation target object included in the manipulation target imageshown in FIG. 6 or the entire manipulation target image in the directionM shown in FIG. 6, as in the case in which it is recognized that theauto scroll manipulation has been performed.

As the image processing apparatus according to the present embodimentseparately sets the display target region and the manipulation targetregion, the apparatus that performs a process related to gesturerecognition can recognize the user manipulation, for example, in theregion including the region not included in the display target region(the region AR1), as shown in the region AR2 shown in FIG. 6, andperform the process according to the recognized user manipulation. Thus,as the image processing apparatus according to the present embodimentseparately sets the display target region and the manipulation targetregion, it is possible to further improve maneuverability of the user.

(1-7) Other Examples of Region Setting Process

Further, the region setting process in the image processing apparatusaccording to the present embodiment is not limited to each of theabove-described processes. For example, when the manipulation targetregion is updated according to the position of the face or the hand, theuser may move out of a range of recognition (e.g., outside an angle ofview of a camera). Here, if the user moves out of the range of therecognition, it is difficult for the apparatus that performs a processrelated to gesture recognition to recognize a gesture and perform theprocess even when the user performs the gesture. Therefore, the imageprocessing apparatus according to the present embodiment notifies theuser that the user is out of the range of recognition.

FIG. 7 is an illustrative diagram illustrating another example of theregion setting process according to the present embodiment. Here, eachof A1, B1, C1, D1, and E1 shown in FIG. 7 illustrates a captured imageand a manipulation target region set in the captured image. Further, A2shown in FIG. 7 illustrates an image of a display target regioncorresponding to the manipulation target region shown in A1 of FIG. 7.Similarly, B2, C2, D2 and E2 shown in FIG. 7 illustrate images ofdisplay target regions corresponding to the manipulation target regionsshown in B1, C1, D1, and E1 of FIG. 7, respectively. Further, FIG. 7illustrates a change in a position of the set manipulation target regionin time series in order of A1 shown in FIG. 7, B1 shown in FIG. 7, C1shown in FIG. 7, D1 shown in FIG. 7, and E1 shown in FIG. 7.

For example, as shown in FIG. 7, when the manipulation target region isclose to the vicinity of an edge of a range of recognition (e.g., anangle of view of a camera), the image processing apparatus according tothe present embodiment appropriately sets the manipulation target regionto move in a direction away from the edge of the range of recognitionthrough animation display as if the manipulation target region hasbumped into a wall and then bounced off the wall. Further, for example,as shown in C1 and C2 of FIG. 7, the image processing apparatusaccording to the present embodiment notifies the user that the user isclose to the outside of the range of recognition by sandwiching a blackimage in a region corresponding to the outside of the range ofrecognition in the manipulation target region. By appropriately settingthe manipulation target region so that the animation display, forexample, as shown in FIG. 7 is realized, the image processing apparatusaccording to the present embodiment can cause the user to intuitivelyrecognize that the user is close to the outside of the range ofrecognition. Further, by appropriately setting the manipulation targetregion so that the animation display, for example, as shown in FIG. 7 isrealized, the image processing apparatus according to the presentembodiment can cause the user to intuitively recognize that the user isclose to the outside of the range of recognition without performingwarning display.

Further, the process for performing a notification related to theoutside of the range of recognition in the image processing apparatusaccording to the present embodiment is not limited to the above process.For example, the image processing apparatus according to the presentembodiment may notify the user that the user is close to the outside ofthe range of recognition by appropriately setting the manipulationtarget region so that another animation display such as animationdisplay showing returning to a wall is realized. Further, a method ofnotifying the user that the user is closer to the outside of the rangeof recognition in the manipulation target region in the image processingapparatus according to the present embodiment is not limited tosandwiching the black region in the image corresponding to the outsideof the range of recognition in the manipulation target region, forexample, as shown in C2 of FIG. 7.

FIG. 8 is an illustrative diagram illustrating an example of a method ofnotifying a user that the user is closer to the outside of a range ofrecognition in the manipulation target region according to the presentembodiment. For example, as shown in A and B of FIG. 8, the imageprocessing apparatus according to the present embodiment may superimposea word or a symbol indicating a warning on a black image sandwiched in aregion corresponding to the outside of the range of recognition in themanipulation target region. Further, for example, as shown in C of FIG.8, the image processing apparatus according to the present embodimentmay sandwich an image other than the black image, such as an image witha wallpaper of an application or with an affordance indicating that itis difficult to go to an end any more, in the region corresponding tothe outside of the range of recognition in the manipulation targetregion. Further, for example, as shown in D of FIG. 8, the imageprocessing apparatus according to the present embodiment may superimposea word or symbol indicating a warning on an image other than the blackimage sandwiched in a region corresponding to the outside of the rangeof recognition in the manipulation target region.

(2) Image Processing

The image processing apparatus according to the present embodimentcombines an image corresponding to the display target region in thecaptured image with the manipulation target image.

Further, image processing in the image processing apparatus according tothe present embodiment is not limited to simple combination of the imagecorresponding to the display target region in the captured image withthe manipulation target image.

When the captured image and the manipulation target image are simplycombined and displayed on the display screen, for example, content ofthe captured image (more strictly, content of the image corresponding tothe display target region) is superimposed on the manipulation targetobject included in the manipulation target image, and accordingly, thereis the possibility of decreasing the visibility of the manipulationtarget object. Therefore, in order to reduce the possibility ofdecreasing visibility of the manipulation target object as describedabove, the image processing apparatus according to the presentembodiment can change the degree of visualization of the captured image,for example, based on the captured image or the manipulation targetimage, and combine an image corresponding to the display target regionin the captured image whose degree of visualization has been changed,with the manipulation target image.

Here, an example of a change in the degree of visualization of thecaptured image according to the present embodiment may include changingthe partial range to partially visualize the captured image or changingthe degree of smoothing to smooth the captured image. Further, a changein the degree of visualization of the captured image according to thepresent embodiment is not limited to the above change. The imageprocessing apparatus according to the present embodiment may change thedegree of visualization of the captured image, for example, byperforming a variety of image processing, such as posterization, mosaic,line drawing, or silhouette display, on the captured image. Further, anexample of partial visualization of the captured image according to thepresent embodiment may include changing a degree of transparency of thecaptured image.

Hereinafter, an example of image processing according to the presentembodiment will be described using a primary example of a case in whichthe image processing apparatus according to the present embodimentcombines the image corresponding to the display target region in thecaptured image whose degree of visualization has been changed with themanipulation target image.

(2-1) First Example of Process According to Change in Partial Range

The image processing apparatus according to the present embodimentdetermines a partial range, for example, based on a predetermined targetto be recognized from the captured image. Also, the image processingapparatus according to the present embodiment, for example, changes adegree of transparency of the captured image based on the determinedpartial range to partially visualize the captured image.

Here, an example of the predetermined target according to the presentembodiment may include an object such as a face of a user or a hand ofthe user. The image processing apparatus according to the presentembodiment, for example, detects a characteristic point such as an eye,a nose, a mouth, and a skeleton of the user or detects a region similarto a luminance distribution and a structure pattern of the face from thecaptured image to recognize a face region including the face of theuser. Further, the image processing apparatus according to the presentembodiment, for example, performs a variety of image processing, such asedge detection or pattern matching, on the captured image to recognizean object of a recognition target, such as a user's hand, from thecaptured image.

Further, a process of recognizing a predetermined target in the imageprocessing apparatus according to the present embodiment is not limitedto the above process. For example, the image processing apparatusaccording to the present embodiment may recognize a moving body as thepredetermined target from the captured image and determine a regionincluding the recognized moving body as the partial range. Here, theimage processing apparatus according to the present embodiment detectsthe moving body, for example, by performing detection of a differencebetween an image corresponding to a current frame and an imagecorresponding to a previous frame or detection of a change in a motionvector, but a method of detecting the moving body is not limitedthereto.

Hereinafter, an example of the process according to a change in thepartial range according to the present embodiment will be described ingreater detail using an example of a process in a case in which a faceis recognized as a predetermined target.

When the face is recognized as the predetermined target, the imageprocessing apparatus according to the present embodiment recognizes theface from the captured image and determines a portion other than therecognized face as the partial range.

FIGS. 9A, 9B, 10A, and 10B are illustrative diagrams illustrating anexample of a combined image displayed in a display screen by the imageprocessing apparatus according to the present embodiment. Here, FIG. 9Ais a conceptual diagram of an actual example shown in FIG. 9B, and FIG.10A is a conceptual diagram of an actual example shown in FIG. 10B.Further, FIGS. 9A, 9B, 10A, and 10B illustrate an example of the imagewhen the image processing apparatus according to the present embodimentdetermines the portion other than the face recognized from the capturedimage, as the partial range.

The image processing apparatus according to the present embodiment, forexample, realizes the visualization of the portion other than therecognized face by performing a process of applying translucentgradation from the recognized face region. Here, the image processingapparatus according to the present embodiment may dynamically change thedegree of transparency when applying the translucent gradation,according to a background portion of the captured image (e.g., theportion other than the recognized face region).

The image processing apparatus according to the present embodiment, forexample, detects a spatial frequency or tint in a background portion ofthe captured image and dynamically changes a transparency degreeaccording to a detection result. For example, the image processingapparatus according to the present embodiment reduces the transparencydegree as the detected spatial frequency is higher (as the image is morecomplex) or reduces the transparency degree as the detected tint isdarker. For example, since the transparency degree is reduced as thedetected spatial frequency is higher or the transparency degree isreduced as the detected tint is darker as described above, the imageprocessing apparatus according to the present embodiment can furtherprevent a decrease in visibility of the manipulation target objectincluded in the manipulation target image.

Further, a process of dynamically changing the degree of transparency inthe image processing apparatus according to the present embodiment isnot limited to the above process. For example, the image processingapparatus according to the present embodiment may identify a position ofthe manipulation target object from the manipulation target image (ordata constituting the manipulation target image, such as HTML data orXML data) and reduce the degree of transparency on a region includingthe identified position. In other words, the image processing apparatusaccording to the present embodiment can also change a partial range inwhich the captured image is partially visualized by determining thepartial range based on the manipulation target object included in themanipulation target image.

Here, the image processing apparatus according to the present embodimentmay exclusively perform one of the process of changing the partial rangebased on the manipulation target object included in the manipulationtarget image and the process of changing the partial range based on thepredetermined target recognized from the captured image or may performboth of the processes. Further, when the image processing apparatusaccording to the present embodiment performs both of the processes, theimage processing apparatus may set a priority for each process, such assetting a priority for the process of changing the partial range basedon the manipulation target object included in the manipulation targetimage to be higher.

For example, as described above, the image processing apparatusaccording to the present embodiment can prevent a decrease in visibilityof the manipulation target object included in the manipulation targetimage by dynamically changing the degree of transparency based on theposition of the manipulation target object.

Further, while FIGS. 9A to 10B illustrate the case in which one user isincluded in the captured image, a plurality of users may be included inthe captured image according to the present embodiment. The imageprocessing apparatus according to the present embodiment sets themanipulation target region, for example, when it is determined that astart condition of gesture recognition is satisfied in the process (1)(region setting process).

(2-2) Second Example of Process According to Change in the Partial Range

In the above description, the example in which the image processingapparatus according to the present embodiment, for example, determinesthe partial range based on the predetermined target recognized from thecaptured image and partially visualizes the captured image based on thedetermined partial range has been shown as the first example of theprocess according to the change in the partial range. However, theprocess according to the change in the partial range in the imageprocessing apparatus according to the present embodiment is not limitedto the above example. For example, the image processing apparatusaccording to the present embodiment can also change the partial range inwhich the captured image is partially visualized, by changing a displayarea of the captured image.

FIG. 11 is an illustrative diagram illustrating an example of a combinedimage displayed on a display screen by the image processing apparatusaccording to the present embodiment. Here, FIG. 11 illustrates anexample in which the partial range in which the captured image isvisualized is changed by applying a mask to the captured image.

The image processing apparatus according to the present embodimentreduces the display area of the captured image by applying a mask to thecaptured image and changes a partial range in which the captured imageis visualized, for example, as shown in FIG. 11.

Further, the process in the image processing apparatus according to thepresent embodiment is not limited to the example shown in FIG. 11. Forexample, the example in which the image processing apparatus accordingto the present embodiment applies the uniform mask to the entirecaptured image is illustrated in FIG. 11, but the image processingapparatus according to the present embodiment may dynamically change,for example, a density, shape or the like of the mask. For example, theimage processing apparatus according to the present embodiment mayidentify a position of the manipulation target object from themanipulation target image (or data constituting the manipulation targetimage, such as HTML data or XML data) and reduce the density of the maskon a region including the identified position or near the region. Inother words, the image processing apparatus according to the presentembodiment can also change a partial range in which the captured imageis partially visualized, by determining the partial range based on themanipulation target object included in the manipulation target image.

For example, as described above, the image processing apparatusaccording to the present embodiment can further prevent a decrease invisibility of the manipulation target object included in themanipulation target image by dynamically changing the density, shape orthe like of the mask based on the position of the manipulation targetobject.

(2-3) Process According to Change in Degree of Smoothing to SmoothCaptured Image

In the above description, the process according to a change in thepartial range has been shown as the process according to the imageprocessing method in the image processing apparatus according to thepresent embodiment, but the process according to the image processingmethod according to the present embodiment is not limited thereto. Forexample, the image processing apparatus according to the presentembodiment may change the degree of smoothing to smooth the capturedimage, as the process of changing the degree of visualization.

FIG. 12 is an illustrative diagram illustrating an example of thecombined image displayed on a display screen by the image processingapparatus according to the present embodiment. Here, FIG. 12 illustratesan example of an image obtained by giving a blur effect to the capturedimage by smoothing the captured image. Further, an example in which, forexample, the manipulation target image including manipulation targetobjects such as “TV,” “Web Browser,” . . . , “Help Guide” and the imagein the display target region have been combined is illustrated in FIG.12.

The image processing apparatus according to the present embodiment givesthe blur effect to the captured image, for example, by applying aGaussian filter to the captured image. Further, the image processingapparatus according to the present embodiment may use any technologycapable of giving the blur effect to the captured image.

Further, the image processing apparatus according to the presentembodiment may combine the manipulation target object with the image inthe display target region, for example, as shown in A of FIG. 12 or maynot combine the manipulation target object with the image in the displaytarget region, for example, as shown in B of FIG. 12. Here, FIG. 12illustrates an example in which the image processing apparatus accordingto the present embodiment combines the manipulation target object withthe image in the display target region so that the image in the displaytarget region is represented with transparency. Whether the manipulationtarget object and the image in the display target region are combined asshown in A of FIG. 12 or the manipulation target object and the image inthe display target region are not combined as shown in B of FIG. 12, theapparatus that performs a process related to gesture recognitionrecognizes a gesture manipulation with respect to the manipulationtarget object and performs a process associated with the manipulationtarget object. For example, if a user (a manipulator) performs a gestureof striking (touching) a desired manipulation target object by hand (anexample of the gesture) on the desired manipulation target object, theapparatus that performs a process related to gesture recognitionrecognizes, for example, that an execution manipulation causing theapparatus that performs a process related to gesture recognition toexecute a process corresponding to the manipulation target object hasbeen performed. Also, when the execution manipulation is recognized tohave been performed, the apparatus that performs a process related togesture recognition performs, for example, a process associated with themanipulation target object of the target on which the executionmanipulation has been performed.

The image processing apparatus according to the present embodimentselectively combines the manipulation target object included in themanipulation target image with the image in the display target region,for example, according to a type or an attribute of the manipulationtarget object included in the manipulation target image. The imageprocessing apparatus according to the present embodiment can reduce thepossibility of decreasing visibility of the manipulation target object,for example, by the combination, by selectively combining themanipulation target object included in the manipulation target imagewith the image in the display target region according to the type orattribute of the manipulation target object included in the manipulationtarget image.

Here, the image processing apparatus according to the present embodimentidentifies a type or attribute of the manipulation target object basedon meta-data corresponding to the manipulation target object or dataconstituting the manipulation target image (e.g., HTML data or XMLdata). Further, an example of a manipulation target object that theimage processing apparatus according to the present embodiment does notcombine with the image of the display target region may include athumbnail or an icon indicating content (e.g., a thumbnail or an iconindicating a still image or a moving image) or a window indicatingcontent being played (e.g., a playback window of a moving image).Further, an example in which both of a manipulation target object to becombined with the image of the display target region and a manipulationtarget object not combined with the image of the display target regionare included in the manipulation target image is illustrated in FIG. 12,but the manipulation target image according to the present embodiment isnot limited to the example shown in FIG. 12. For example, themanipulation target image according to the present embodiment mayinclude only one of the manipulation target object to be combined withthe image of the display target region and the manipulation targetobject not combined with the image of the display target region.Further, it is understood that an example of the manipulation targetobject not combined with the image of the display target regionaccording to the present embodiment is not limited to the example shownabove, and the image processing apparatus according to the embodimentcan combine the thumbnail or the like indicating the content shown abovewith the image of the display target region.

Further, the object included in the manipulation target image that theimage processing apparatus according to the present embodimentselectively combines with the image of the display target region is notlimited to the manipulation target object included in the manipulationtarget image. For example, according to a type or an attribute of anobject that is not the manipulation target object included in themanipulation target image (which may be hereinafter referred to as a“non-manipulation target object”), the image processing apparatusaccording to the present embodiment may selectively combine thenon-manipulation target object included in the manipulation target imagewith the image of the display target region, as in a case in which themanipulation target object included in the manipulation target image isselectively combined. By selectively combining the non-manipulationtarget object included in the manipulation target image and the image ofthe display target region according to the type or the attribute of thenon-manipulation target object included in the manipulation targetimage, the image processing apparatus according to the presentembodiment, for example, can reduce the possibility of decreasingvisibility of the non-manipulation target object by the combination.Here, an example of a non-manipulation target object that the imageprocessing apparatus according to the present embodiment does notcombine with the image of the display target region may include a textdescription window for content, which describes the content using text.

Further, an effect given to the captured image by the image processingapparatus according to the present embodiment smoothing the capturedimage is not limited to the blur effect. For example, as the imageprocessing apparatus according to the present embodiment smooths thecaptured image, a painterly effect (e.g., a brush stroke effect) can begiven to the captured image, as if it were painted by a painter.

If a painterly effect is given to the captured image, the imageprocessing apparatus according to the present embodiment, for example,calculates a vector direction of an edge in the image (or an anglebetween the vector and a reference direction such as a verticaldirection or a horizontal direction of the image) and adjusts intensityof an edge-preservation smoothing filter based on the vector direction.Also, the image processing apparatus according to the present embodimentsmooths the captured image using the adjusted edge-preservationsmoothing filter to give the painterly effect to the captured image.Here, an example of the edge-preservation smoothing filter according tothe present embodiment may include a smoothing filter capable ofpreserving an edge, such as a bilateral filter. A process of giving thepainterly effect to the captured image in the image processing apparatusaccording to the present embodiment is not limited to the above process.The image processing apparatus according to the present embodiment mayuse any technology capable of giving the painterly effect to thecaptured image.

The image processing apparatus according to the present embodimentperforms, for example, any of the processes shown in (2-1) to (2-3) inthe process (2) (image processing) to change the degree of visualizationof the captured image and combines the image corresponding to thedisplay target region in the captured image whose degree ofvisualization has been changed with the manipulation target image. Here,the image processing apparatus according to the present embodiment, forexample, may perform a process defined in advance or may perform theprocess selected by the user.

Further, the process of combining the image corresponding to the displaytarget region in the captured image whose degree of visualization hasbeen changed with the manipulation target image in the process (2)(image processing) according to the present embodiment is not limited tothe process shown in (2-1) to (2-3).

For example, a region in which a gesture can be recognized and a regionin which a gesture is not recognized may be included in the manipulationtarget image according to the present embodiment. When the region inwhich a gesture can be recognized and the region in which a gesture isnot recognized are included in the manipulation target image accordingto the present embodiment, the image processing apparatus according tothe present embodiment, for example, may not combine the captured imagein the region in which a gesture is not recognized in the manipulationtarget image.

FIG. 13 is an illustrative diagram illustrating an example of themanipulation target image according to the present embodiment. Forexample, a manipulation target object not suitable for a gesturemanipulation (e.g., a manipulation target object that it is not easy toselect by a gesture manipulation), such as fine links constituting menusas shown in A1 of FIG. 13 (an example of the manipulation targetobject), may be present in the manipulation target image according tothe present embodiment. Further, for example, an object that is not amanipulation target object whose selection manipulation is performed asshown in A2 of FIG. 13 (a non-manipulation target object) may also bepresent in the manipulation target image according to the presentembodiment. Therefore, in the present embodiment, for example, a regionincluding manipulation target objects not suitable for a gesturemanipulation as shown in A of FIG. 13 or a region including objects thatare not manipulation target objects as shown in B of FIG. 13 is a regionin which a gesture is not recognized in the manipulation target image.Further, for example, a region including manipulation target objectsthat can be manipulated by a gesture as shown in B of FIG. 13 in themanipulation target image is a region in which a gesture can berecognized in the manipulation target image.

Here, the image processing apparatus according to the present embodimentidentifies the region in which a gesture is not recognized in themanipulation target image or the region in which a gesture can berecognized in the manipulation target image, for example, based oninformation indicating the region in which a gesture is not recognizedin the manipulation target image associated with the manipulation targetimage (e.g., a meta data or a description of a tag in markup language).Further, a method of specifying the region in which a gesture is notrecognized in the manipulation target image is not limited to the abovemethod. For example, the image processing apparatus according to thepresent embodiment may analyze the manipulation target image and seteach of the region in which a gesture can be recognized and the regionin which a gesture is not recognized.

For example, in the example of FIG. 13, the image processing apparatusaccording to the present embodiment does not combine the captured image(or the captured image whose degree of visualization has been changed)in the regions shown in A1 and A2 of FIG. 13 and combines the capturedimage (or the captured image whose degree of visualization has beenchanged) in the region shown in B of FIG. 13. Here, in FIG. 13, anexample in which the image processing apparatus according to the presentembodiment has combined the captured image whose degree of visualizationhas been changed with the manipulation target image is simplyillustrated.

The image processing apparatus according to the present embodimentcombines the image corresponding to the display target region in thecaptured image with the manipulation target image in the process (2)(image processing). Also, the combined image is displayed on the displayscreen through a process (3) (display control process), which will bedescribed below. Accordingly, as the image processing apparatusaccording to the present embodiment combines the image corresponding tothe display target region with the manipulation target image in theprocess (2) (image processing), difficulty for the user in manipulatingmanipulation target objects arranged within the display screen isprevented.

Further, the image processing apparatus according to the presentembodiment may change the degree of visualization of the captured imagebased on the captured image or the manipulation target image, andcombine an image corresponding to the display target region in thecaptured image whose degree of visualization has been changed with themanipulation target image, for example, as described above. Also, thecombined image is displayed on the display screen through a process (3)(display control process), which will be described below. Accordingly,as the image processing apparatus according to the present embodimentchanges the degree of visualization of the captured image in the process(2) (image processing), it is possible to prevent the visibility of themanipulation target object from decreasing in comparison with the casein which the captured image and the manipulation target image are simplycombined and displayed on the display screen.

Further, the process (2) (image processing) in the image processingapparatus according to the present embodiment is not limited to theabove process.

(2-4) Other Examples of Image Processing (2-4-1) First Example of OtherImage Processing

For example, when the manipulation target region has been set in theprocess (1) (region setting process), the image processing apparatusaccording to the present embodiment may sequentially change the positionand size of the region in the captured image to be combined with themanipulation target image from the position and size of a predeterminedregion larger than the display target region to the position and size ofthe display target region. When the position and size of the region inthe captured image is changed, the image processing apparatus accordingto the present embodiment sequentially combines, for example, the imagecorresponding to the region whose position and size are sequentiallychanged with the manipulation target image.

As the image processing apparatus according to the present embodimentchanges, for example, the position and size of the region in thecaptured image to be combined with the manipulation target image andsequentially combines the image corresponding to the region with themanipulation target image as described above, an image corresponding toa predetermined region is first displayed on the display screen and animage of animation display is displayed, such as zooming of the imagecorresponding to the predetermined region until the region correspondingto the display target region is displayed, through a process (3)(display control process), which will be described below.

Here, a situation in which there is a plurality of users in the capturedimage is assumed. In this situation, if an object that becomes a cluefor identifying an individual, such as a face or clothing is notincluded in the image displayed on the display screen so that the usercan identify the object, it may be difficult for each user to recognizea user whose gesture is currently recognized (in other words, where themanipulation target region is set) from the image displayed on thedisplay screen.

On the other hand, the image processing apparatus according to thepresent embodiment displays, on the display screen, an image ofanimation display such as zooming of the image corresponding to thepredetermined region up to the image corresponding to the display targetregion after the image corresponding to the predetermined region isfirst displayed. That is, each user views the image in the displaytarget region corresponding to the manipulation target region afterviewing an image that has been looked down at once. Thus, even whenthere are a plurality of users in the captured image, the imageprocessing apparatus according to the present embodiment can enable eachuser to more easily recognize a user whose gesture is currentlyrecognized (in other words, where the manipulation target region hasbeen set) from the image displayed on display screen.

FIG. 14 is an illustrative diagram illustrating another example of theimage processing in the image processing apparatus according to thepresent embodiment, and illustrates an example of animation displayaccording to the present embodiment. Here, FIG. 14 shows an example of acaptured image and an image to be displayed on the display screen(however, a manipulation target image is omitted) arranged in timeseries (arranged to the right as time elapses in FIG. 14). Further, inFIG. 14, a captured image A1 and an image A2 to be displayed on thedisplay screen with the captured image are temporally correspondingimages. Similarly, a captured image B1 and an image B2 to be displayedon the display screen with the captured image, a captured image C1 andan image C2 to be displayed on the display screen with the capturedimage, a captured image D1 and an image D2 to be displayed on thedisplay screen with the captured image, and a captured image E1 and animage E2 to be displayed on the display screen with the captured imageare temporally corresponding images. Further, in FIG. 14, the displaytarget region is indicated by “AR_T” and the predetermined region isindicated by “AR0.” Further, in FIG. 14, the region whose position andsize are sequentially changed is indicated by “AR1.”

Referring to FIG. 14, the image processing apparatus according to thepresent embodiment does not display an image for feedback on the displayscreen before a gesture manipulation starts (A2 shown in FIG. 14). Inthe process (1) (region setting process), if it is determined that astart condition of gesture recognition, such as waving goodbye issatisfied (if a gesture recognition start trigger (T shown in FIG. 14)is detected), the image processing apparatus according to the presentembodiment displays the image corresponding to the predetermined regionAR0 on the display screen. Here, although the example in which thepredetermined region AR0 is the entire captured image is illustrated inFIG. 14, the predetermined region larger than the display target regionaccording to the present embodiment is not limited to the above region.

FIG. 15 is an illustrative diagram illustrating an example of thepredetermined region larger than the display target region according tothe present embodiment. As shown in FIG. 15, the image processingapparatus according to the present embodiment may, for example, set arectangular region having any size greater than a display target regionincluding a face region of a user detected from a captured image as thepredetermined region from which zooming starts.

An example of animation display according to another example of theimage processing in the image processing apparatus according to thepresent embodiment will be described with reference to FIG. 14 again.When an image corresponding to the predetermined region AR0 is displayedon the display screen, the image processing apparatus according to thepresent embodiment sequentially changes, over time, the position andsize of the region in the captured image to be combined with themanipulation target image from the position and size of thepredetermined region AR0 to the position and size of the display targetregion AR_T (C1, D1, and E1 shown in FIG. 14). Thus, an image zoomed tothe display target region AR_T will be displayed on the display screenover time, as shown in C2 of FIG. 14 to E2 of FIG. 14.

Here, the apparatus that performs a process related to gesturerecognition performs the process related to the gesture recognition forthe manipulation target region even while zooming, for example, as shownin FIG. 14, but the process in the apparatus that performs a processrelated to gesture recognition is not limited to the above process. Forexample, the apparatus that performs a process related to gesturerecognition may not perform the process related to the gesturerecognition while zooming as shown in FIG. 14. As the apparatus thatperforms a process related to gesture recognition does not perform theprocess related to the gesture recognition while zooming as shown inFIG. 14, it is possible to prevent a wrong manipulation of the user whenit is difficult for the user to recognize the manipulation targetregion.

When the manipulation target region has been set in the process (1)(region setting process), the image processing apparatus according tothe present embodiment displays, on the display screen, an image ofanimation display such as zooming of the image corresponding to thepredetermined region up to the image corresponding to the display targetregion after the image corresponding to the predetermined region isfirst displayed, for example, as shown in FIG. 14. Thus, each user canrecognize who is a manipulator, from the image displayed on the displayscreen.

(2-4-2) Second Example of Other Image Processing

Further, image processing when the manipulation target region has beenset in the process (1) (region setting process) is not limited to theprocess when the manipulation target region has been newly set, forexample as shown in FIG. 14. For example, the manipulation target regionmay have already been set when the manipulation target region was set inthe process (1) (region setting process).

When a new manipulation target region different from the manipulationtarget region that has already been set in the process (1) (regionsetting process) has been set, the image processing apparatus accordingto the present embodiment, for example, may sequentially change theposition and size of the region in the captured image to be combinedwith the manipulation target image from the position and size of thedisplay target region corresponding to the manipulation target regionthat has already been set to the position and size of a predeterminedregion larger than the display target region. Further, when the positionand size of the region in the captured image has been changed to theposition and size of the predetermined region, the image processingapparatus according to the present embodiment, for example, sequentiallychanges the position and size of the region in the captured image fromthe position and size of the predetermined region to the position andsize of the display target region corresponding to the newly setmanipulation target region. When the position and size of the region inthe captured image are changed, the image processing apparatus accordingto the present embodiment, for example, sequentially combines the imagecorresponding to the region whose position and size are sequentiallychanged with the manipulation target image.

As the image processing apparatus according to the present embodimentchanges the position and size of the region in the captured image to becombined with the manipulation target image, for example, as describedabove and sequentially combines an image corresponding to the regionwith the manipulation target image, an image displayed on the displayscreen through a process (3) (display control process), which will bedescribed below is an image of animation display, for example, as shownin the following (i) and (ii).

(i) Animation display, such as zooming, of an image corresponding to thedisplay target region corresponding to the manipulation target regionthat has already been set, until an image corresponding to apredetermined region is displayed (zoom-out display)

(ii) Animation display, such as zooming, of the image corresponding tothe predetermined region until the image corresponding to the displaytarget region is displayed (zoom-in display)

Here, a situation in which there are a plurality of users in thecaptured image is assumed. In the above situation, if a display targetregion to be displayed on the display screen is instantaneously switchedfrom a display target region corresponding to a manipulation targetregion set for one user to a display target region corresponding to amanipulation target region set for the other user, each user is likelyto be unable to recognize a user whose gesture is currently recognized(in other words, where the manipulation target region has been set) fromthe image displayed on the display screen.

On the other hand, the image processing apparatus according to thepresent embodiment first displays, on the display screen, an image ofanimation display, such as zooming of the image of the display targetregion corresponding to the manipulation target region that has alreadybeen set, up to the image of the predetermined region. Also, the imageprocessing apparatus according to the present embodiment displays, onthe display screen, an image of animation display, such as zooming ofthe image of the predetermined region up to the image corresponding tothe display target region corresponding to the newly set manipulationtarget region. That is, each user views the image of the display targetregion corresponding to the newly set manipulation target region throughzoom-in display after viewing an image that has been looked down at oncethrough zoom-out display. Thus, even when there are a plurality of usersin the captured image and a user corresponding to the set manipulationtarget region is switched, the image processing apparatus according tothe present embodiment can enable each user to more easily recognize auser whose gesture is currently recognized (in other words, where themanipulation target region has been set) from the image displayed on thedisplay screen.

FIG. 16 is an illustrative diagram illustrating another example of theimage processing in the image processing apparatus according to thepresent embodiment, and illustrates an example of the animation displayaccording to the present embodiment. Here, FIG. 16 illustrates anexample of a captured image and an image displayed on the display screen(however, the manipulation target image is omitted) arranged in timeseries (in FIG. 16, arranged to the right as time elapses). Further, inFIG. 16, a captured image A1 and an image A2 displayed on the displayscreen with the captured image are temporally corresponding images.Similarly, a captured image B1 and an image B2 displayed on the displayscreen with the captured image, a captured image C1 and an image C2displayed on the display screen with the captured image, a capturedimage D1 and an image D2 displayed on the display screen with thecaptured image, and a captured image E1 and an image E2 displayed on thedisplay screen with the captured image are temporally correspondingimages. Further, in FIG. 16, a display target region corresponding tothe manipulation target region that has already been set is indicated by“AR_T1” and a display target region corresponding to the manipulationtarget region that has been newly set is indicated by “AR_T2.” Further,in FIG. 16, the predetermined region is indicated by “AR0,” and regionswhose position and size are changed sequentially are indicated by “AR1”and “AR2.”

If a new manipulation target region is set when the display targetregion AR_T1 has been displayed on the display screen (A1 and A2 shownin FIG. 16), the image processing apparatus according to the presentembodiment sequentially changes a position and size of the region in thecaptured image to be combined with the manipulation target image from aposition and a size of the display target region AR_T1 to a position andsize of the predetermined region AR0 over time (B1 and C1 shown in FIG.16). Thus, an image zoomed to the predetermined region AR0 over time isdisplayed on the display screen, as shown in B2 and C2 of FIG. 16. Here,although the example in which the predetermined region AR0 is the entirecaptured image has been illustrated in FIG. 16, a predetermined regionlarger than the display target region according to the presentembodiment is not limited to the above region, as described above.

Further, if the position and size of the region in the captured image tobe combined with the manipulation target image are changed to theposition and size of the predetermined region AR0, the image processingapparatus according to the present embodiment sequentially changes aposition and size of the region in the captured image to be combinedwith the manipulation target image from the position and size of thepredetermined region AR0 to the position and size of the display targetregion AR_T2 over time (D1 and E1 shown in FIG. 16). Thus, an imagezoomed to the display target region AR_T2 over time is displayed on thedisplay screen, as shown in D2 and E2 of FIG. 16.

Here, the apparatus that performs a process related to gesturerecognition performs the process related to the gesture recognition onthe manipulation target region even while zooming, for example, as shownin FIG. 16, but the process in the apparatus that performs a processrelated to gesture recognition is not limited thereto. For example, theapparatus that performs a process related to gesture recognition may notperform the process related to the gesture during zooming as shown inFIG. 16, like during zooming as shown in FIG. 14.

FIG. 17 is an illustrative diagram illustrating another example of theimage processing in the image processing apparatus according to thepresent embodiment. FIG. 17 illustrates an example of a change in aregion of the captured image included in an image displayed on thedisplay screen when animation display according to the presentembodiment is performed. For example, as shown in FIG. 17, the imageprocessing apparatus according to the present embodiment changes theregion displayed on the display screen from the region of the entirecaptured image to realize, for example, animation display in which theanimation display shown in (i) and the animation display shown in (ii)are combined.

If a new manipulation target region different from the manipulationtarget region that has already been set has been set in the process (1)(region setting process), the image processing apparatus according tothe present embodiment displays, on the display screen, an image ofanimation display in which zoom-out display from the display targetregion corresponding to the manipulation target region that has alreadybeen set and zoom-in display to the display target region correspondingto the manipulation target region that has been newly set are combined,for example, as shown in FIG. 16. Therefore, each user can recognize achange in the manipulator from the image displayed on the displayscreen.

Further, the image processing apparatus according to the presentembodiment may adjust the degree of visualization of the manipulationtarget image, for example, during the zooming animation display. Anexample of adjusting the degree of visualization of the manipulationtarget image in the image processing apparatus according to the presentembodiment may include decreasing a degree of transparency of a GUI suchas a widget. By lowering the degree of transparency of the GUI, theimage processing apparatus according to the present embodiment cantemporarily increase visibility of the image (a feedback image) of theregion in the captured image to be combined with the manipulation targetimage and cause the user to more easily recognize switching of themanipulator. Further, the example of adjusting the degree ofvisualization of the manipulation target image in the image processingapparatus according to the present embodiment is not limited to theabove example. For example, the image processing apparatus according tothe present embodiment may not temporarily display a GUI such as awidget.

Further, for example, when the image processing apparatus according tothe present embodiment changes the degree of visualization of thecaptured image, the image processing apparatus may adjust the degree ofvisualization of the captured image. An example of adjusting the degreeof visualization of the captured image in the image processing apparatusaccording to the present embodiment may include weakening blur additionand subtraction of the captured image.

(2-4-3) Third Example of Other Image Processing

Further, image processing in a case in which the new manipulation targetregion different from the manipulation target region that has alreadybeen set in the process (1) (region setting process) has been set is notlimited to the process related to realization of the animation displayin which zoom-out display and zoom-in display are combined, for example,as shown in FIG. 16.

For example, if a new manipulation target region different from themanipulation target region that has already been set in the process (1)(region setting process) has been set, the image processing apparatusaccording to the present embodiment may change the location of theregion in the captured image to be combined with the manipulation targetimage to move from the display target region corresponding to themanipulation target region that has already been set to the displaytarget region corresponding to the newly set manipulation target region.

FIG. 18 is an illustrative diagram illustrating another example of theimage processing in the image processing apparatus according to thepresent embodiment, and illustrates an example of a movement of a regiondisplayed on the display screen according to the present embodiment.Here, FIG. 18 illustrates the captured image. Further, in FIG. 18, thedisplay target region corresponding to the manipulation target regionthat has already been set is indicated by “AR_T1” and the display targetregion corresponding to the manipulation target region that has beennewly set is indicated by “AR_T2.” Further, in FIG. 18, a trajectory ofthe movement of the region is indicated by “L.”

As shown in L of FIG. 18, the image processing apparatus according tothe present embodiment, for example, changes the position of the regionin the captured image to be combined with the manipulation target imageto draw a trajectory passing through the face region recognized from thecaptured image. As the position of the region in the captured image tobe combined with the manipulation target image is changed to draw thetrajectory passing through the face region recognized from the capturedimage, for example, as shown in FIG. 18, display in which switching of amanipulator is highlighted is realized.

Further, a method of changing the position of the region in the capturedimage to be combined with the manipulation target image according to thepresent embodiment is not limited to the above method. For example, theimage processing apparatus according to the present embodiment maychange the position of the region in the captured image to be combinedwith the manipulation target image, to move, at the shortest distance,from the display target region corresponding to the manipulation targetregion that has already been set, to the display target regioncorresponding to the manipulation target region that has been newly set.

(3) Display Control Process

If the process (2) (image processing) is completed, the image processingapparatus according to the present embodiment displays the combinedimage on the display screen.

Here, when the image processing apparatus according to the presentembodiment displays the image on a display screen of an external displaydevice, the image processing apparatus according to the presentembodiment transmits, for example, an image signal indicating the imageto the display device to display the image on the display device.Further, when the image processing apparatus according to the presentembodiment displays the image on a display screen of a display unit(which will be described below), the image processing apparatusaccording to the present embodiment delivers an image signal indicatingthe image to the display unit (which will be described below) to displaythe image.

The image processing apparatus according to the present embodimentperforms, for example, the process (1) (region setting process), theprocess (2) (image processing), and the process (3) (display controlprocess), as the processes according to the image processing methodaccording to the present embodiment. In the process (1) (region settingprocess), the image processing apparatus according to the presentembodiment sets the manipulation target region. Further, in the process(2) (image processing), the image processing apparatus according to thepresent embodiment combines the image corresponding to the displaytarget region in the captured image (the region corresponding to themanipulation target region) with the manipulation target image. Also,the combined image is displayed on the display screen through theprocess (3) (display control process).

Here, since the image processing apparatus according to the presentembodiment combines the image corresponding to the display target regioncorresponding to the set manipulation target region with themanipulation target image in the process (2) (image processing),difficulty for the user in manipulating manipulation target objectsarranged within the display screen is prevented.

Accordingly, the image processing apparatus according to the presentembodiment can display the captured image and the manipulation targetimage on the display screen while preventing degradation of themaneuverability of the user.

Further, the image processing apparatus according to the presentembodiment may change the degree of visualization of the captured imagein the process (2) (image processing). When the degree of visualizationof the captured image is changed in the process (2) (image processing),the image processing apparatus according to the present embodiment canprevent a decrease in the visibility of the manipulation target objectin comparison with the case in which the captured image and themanipulation target image are simply combined and displayed on thedisplay screen. Accordingly, if the degree of visualization of thecaptured image is changed in the process (2) (image processing), theimage processing apparatus according to the present embodiment candisplay the captured image and the manipulation target image on thedisplay screen while preventing a decrease in the visibility of themanipulation target object and preventing degradation of maneuverabilityof the user.

Further, if the image processing apparatus according to the presentembodiment determines the partial range in which the captured image ispartially visualized, based on the manipulation target object includedin the manipulation target image in the process (2) (image processing),it is possible to display an image in which the manipulation targetobject included in the manipulation target image is more clearlydisplayed, on the display screen. Accordingly, in the above case, theimage processing apparatus according to the present embodiment canfurther improve, for example, the visibility of the manipulation targetobject included in the manipulation target image (i.e., a manipulationtarget object constituting a GUI).

Further, if the image processing apparatus according to the presentembodiment determines the partial range in which the captured image ispartially visualized, based on a predetermined target recognized fromthe captured image in the process (2) (image processing), a face of theuser performing the manipulation is not displayed on the display screen,for example, as shown in FIGS. 9A to 10B. Accordingly, in the abovecase, the image processing apparatus according to the present embodimentcan realize, for example, a mirror image representation causing the usernot to feel vividness, thereby feeding the gesture recognition causingthe user not to feel vividness back to the user.

Further, since the image processing apparatus according to the presentembodiment changes the degree of visualization of the captured image inthe process (2) (image processing), it is possible to control thevisualization of a region unnecessary for a gesture-based manipulationin comparison with a case in which the captured image and themanipulation target image are simply combined and displayed on thedisplay screen. Accordingly, the image processing apparatus according tothe present embodiment, for example, can further facilitate a finergesture-based manipulation by the user, thereby further improvingmaneuverability of the user.

Further, the image processing apparatus according to the presentembodiment can display the image in the display target regioncorresponding to the manipulation target region on the display screenthrough the animation display, such as zooming, for example, through theprocess (2) (image processing) and the process (3) (display controlprocess). By displaying the image in the display target regioncorresponding to the manipulation target region on the display screenthrough the animation display such as zooming, the image processingapparatus according to the present embodiment can cause the user to morereadily recognize, from the image displayed on the display screen, auser whose gesture is currently recognized.

Further, the process according to the image processing method accordingto the present embodiment in the image processing apparatus according tothe present embodiment is not limited to the process (1) (region settingprocess) to the process (3) (display control process). For example, theimage processing apparatus according to the present embodiment maynotify the user of a gesture manipulation possible in the manipulationtarget object based on the position of the recognition target object onwhich the gesture recognition is performed and the position of themanipulation target object manipulated according to a result of thegesture recognition included in the manipulation target image, in thecombined image (notification process).

Here, an example of the recognition target object according to thepresent embodiment may include a gesture recognition target forrecognition of a gesture of a user, such as a part of a body of the usersuch as a hand of the user or a shape of an object manipulated by theuser.

FIGS. 19 to 21 are illustrative diagrams illustrating examples of thenotification process in the image processing apparatus according to thepresent embodiment. Here, each of FIGS. 19 to 21 illustrates an examplein which a gesture manipulation possible with respect to an iconconstituting a GUI or content, which is selected by a hand of a user (anexample of a recognition target object) is visually notified of throughthe notification process according to the present embodiment. Further,in FIGS. 19 to 21, an example in which the image processing apparatusaccording to the present embodiment has combined the captured imagewhose degree of visualization has been changed with the manipulationtarget image is illustrated simply.

For example, FIG. 19 illustrates an example in which the imageprocessing apparatus according to the present embodiment displaysauxiliary icons indicating scrollable directions (I1 and I2 in FIG. 19)on the display screen when the position of the hand of the user ispresent in a range in which an icon can be manipulated.

Further, for example, FIG. 20 illustrates an example in which the imageprocessing apparatus according to the present embodiment displays anicon indicating a gesture by which content can be determined, on thedisplay screen, when the position of the hand of the user is present ina range in which the content can be manipulated. Further, the imageprocessing apparatus according to the present embodiment may notify theuser of the gesture by which the content can be determined, throughanimation display.

Further, for example, FIG. 21 illustrates an example in which an iconindicating a gesture by which the content can be determined or a gestureby which the content can be switched is displayed on the display screenwhen the position of the hand of the user is present in a range in whichthe content can be manipulated. Further, the image processing apparatusaccording to the present embodiment may notify the user of the gestureby which the content can be determined or the gesture by which thecontent can be switched, through animation display.

Further, for example, as shown in FIG. 21, the image processingapparatus according to the present embodiment may notify the userwhether an object can be manipulated by a gesture by changingrepresentations of an object that can be manipulated by a gesture (Ashown in FIG. 21) and objects difficult to be manipulated by a gesture(B to E shown in FIG. 21)). Here, an example of a representing methodindicating whether manipulation can be performed by the gesture mayinclude representation in which the degree of transparency is changed.

The image processing apparatus according to the present embodimentperforms a process of visually notifying of a gesture manipulationpossible in the manipulation target object, for example, as describedabove, as the notification process (4). Further, the notificationprocess in the image processing apparatus according to the presentembodiment is not limited to the process of visually notifying of agesture manipulation possible in the manipulation target object as shownin FIGS. 19 to 21. For example, the image processing apparatus accordingto the present embodiment may notify of the gesture manipulationpossible in the manipulation target object using an auditory notifyingmethod using sound (including music or a beeping sound; the same applieshereinafter). Further, the image processing apparatus according to thepresent embodiment may perform a combination of visual notification andauditory notification.

Here, an example of a target that the image processing apparatusaccording to the present embodiment causes to perform the notificationmay include a display unit (which will be described below) or a soundoutput unit (which will be described below) included in the ownapparatus (the image processing apparatus according to the presentembodiment), and/or an external apparatus, such as an external displaydevice or an external sound output device.

[2] One Example of Process According to Image Processing MethodAccording to Present Embodiment

FIG. 22 is a flow diagram illustrating an example of the processaccording to the image processing method according to the presentembodiment. Hereinafter, the image processing apparatus according to thepresent embodiment will be described as performing the processillustrated in FIG. 22. Here, for example, a process of steps S100,S102, S110, and S112 in FIG. 22 corresponds to the process (1) (regionsetting process). Further, for example, a process of steps S104 to S108and steps S114 and S116 in FIG. 22 corresponds to the process (2) (imageprocessing) and the process (3) (display control process). Further,hereinafter, for example, the combined image will be described as beingdisplayed on the display screen, basically, through the process (2)(image processing) and the process (3) (display control process) untilthe process of step S108 is performed after the process of step S102 isperformed.

The image processing apparatus according to the present embodimentdetermines whether the gesture recognition starts (S100). Here, theimage processing apparatus according to the present embodiment, forexample, determines whether a start condition of gesture recognition(trigger of gesture recognition start) is satisfied, and determines thatthe gesture recognition starts if it is determined that the startcondition of gesture recognition is satisfied.

If it is not determined in step S100 that the gesture recognitionstarts, the image processing apparatus according to the presentembodiment does not proceed with the process until the gesturerecognition is determined to start.

Further, when it is determined in step S100 that the gesture recognitionstarts, the image processing apparatus according to the presentembodiment sets a manipulation target region (S102).

If the manipulation target region is set in step S102, the imageprocessing apparatus according to the present embodiment performs a zoomanimation display (S104). Here, an example of the zoom animation displayin step S104 may include zoom-in display from a predetermined regionlarger than a display target region corresponding to the manipulationtarget region set in step S102 to the display target region.

Further, although not shown in FIG. 22, the apparatus that performs aprocess related to gesture recognition recognizes a gesture in themanipulation target region set in step S102 and performs a processaccording to a recognition result, subsequent to the process of stepS102.

The image processing apparatus according to the present embodimentdetermines whether the gesture recognition in the apparatus thatperforms a process related to gesture recognition has ended (S106). Ifthe apparatus that performs a process related to gesture recognition isthe own apparatus (the image processing apparatus according to thepresent embodiment), the image processing apparatus according to thepresent embodiment, for example, determines whether the end condition ofgesture recognition (trigger of gesture recognition end) is satisfied,and determines that the gesture recognition has ended if it isdetermined that the end condition of gesture recognition is satisfied,as in step S100. Further, the image processing apparatus according tothe present embodiment may determine that the end condition of gesturerecognition is satisfied, for example, if a manipulation or a motion ofthe detection target is not detected for a certain period of time or ifcontent, an icon or the like is selected. Further, if the apparatus thatperforms a process related to gesture recognition is an externalapparatus, the image processing apparatus according to the presentembodiment determines that the gesture recognition has ended, forexample, when receiving a signal indicating that the gesture recognitionhas ended, which is transmitted from the external apparatus.

If it is determined in step S106 that the gesture recognition in theapparatus that performs a process related to gesture recognition hasended, the image processing apparatus according to the presentembodiment displays the manipulation target image on the display screenand ends image-based feedback to the user. Also, the image processingapparatus according to the present embodiment ends the process accordingto the image processing method according to the present embodiment.

Further, if it is not determined in step S106 that the gesturerecognition in the apparatus that performs a process related to gesturerecognition has ended, the image processing apparatus according to thepresent embodiment determines whether new gesture recognition starts(S110). Here, the image processing apparatus according to the presentembodiment determines that the new gesture recognition starts if it isdetermined that the start condition of the new gesture recognition issatisfied, for example, through the same process as step S100. Further,a target on which the new gesture recognition determined in step S110 isperformed, for example, may be a user different from the usercorresponding to the manipulation target region which has already beenset or may be the same user as the user corresponding to themanipulation target region which has already been set.

If it is not determined in step S110 that the new gesture recognitionstarts, the image processing apparatus according to the presentembodiment repeats the process from step S106.

Further, if it is determined in step S110 that the new gesturerecognition starts, the image processing apparatus according to thepresent embodiment sets the manipulation target region (S112), as instep S102.

If the manipulation target region is set in step S112, the imageprocessing apparatus according to the present embodiment performs zoomanimation display (S114 and S116). Also, the image processing apparatusaccording to the present embodiment repeats the process from step S106.Here, an example of the zoom animation display in step S114 may includezoom-out display from the display target region corresponding to themanipulation target region which has already been set, to apredetermined region larger than the display target region and a displaytarget region corresponding to the manipulation target region newly setin step S112. Further, an example of the zoom animation display in stepS116 may include zoom-in display from the predetermined region to thedisplay target region corresponding to the manipulation target regionnewly set in step S112.

The image processing apparatus according to the present embodimentperforms, for example, the process illustrated in FIG. 22 to realize theprocess (1) (region setting process) to the process (3) (display controlprocess) according to the image processing method according to thepresent embodiment. Further, it is understood that the process accordingto the image processing method according to the present embodiment isnot limited to the process illustrated in FIG. 22.

Image Processing Apparatus According to the Present Embodiment

Next, an example of a configuration of the image processing apparatusaccording to the present embodiment capable of performing the processaccording to the image processing method according to the presentembodiment described above will be described.

FIG. 23 is a block diagram illustrating an example of a configuration ofthe image processing apparatus 100 according to the present embodiment.For example, the image processing apparatus 100 includes a communicationunit 102 and a control unit 104.

Further, the image processing apparatus 100 may include, for example, aROM (Read Only Memory; not shown), a RAM (Random Access Memory; notshown), a storage unit (not shown), a manipulation unit (not shown) thatcan be manipulated by the user, and a display unit (not shown) thatdisplays various screens on a display screen. In the image processingapparatus 100, for example, the respective components are connected by abus as a data transmission line.

Here, the ROM (not shown) stores a program or data for control such ascalculation parameters, which is used by the control unit 104. The RAM(not shown) temporarily stores, for example, a program to be executed bythe control unit 104.

The storage unit (not shown) is a storage medium included in the imageprocessing apparatus 100, and stores, for example, various data such asdata constituting the manipulation target image or applications. Here,examples of the storage unit (not shown) may include a magneticrecording medium such as a hard disk, and a non-volatile memory such asan EEPROM (Electrically Erasable and Programmable Read Only Memory) or aflash memory. Further, the storage unit (not shown) may be detachablefrom the image processing apparatus 100. Further, the manipulation unit(not shown) may include a manipulation input device, which will bedescribed below, and the display unit (not shown) may include a displaydevice, which will be described below.

Example of Hardware Configuration of Image Processing Apparatus 100

FIG. 24 is an illustrative diagram illustrating an example of a hardwareconfiguration of the image processing apparatus 100 according to thepresent embodiment. The image processing apparatus 100 includes, forexample, an MPU 150, a ROM 152, a RAM 154, a recording medium 156, aninput and output interface 158, a manipulation input device 160, adisplay device 162, and a communication interface 164. Further, in theimage processing apparatus 100, the respective components are connected,for example, by a bus 166 as a data transmission path.

The MPU 150, for example, functions as the control unit 104 thatincludes an MPU (Micro Processing Unit) or various processing circuits,and controls the entire image processing apparatus 100. Further, in theimage processing apparatus 100, the MPU 150 plays the role of, forexample, a region setting unit 110, an image processing unit 112, and adisplay control unit 114, which will be described below.

The ROM 152 stores a program or data for control, such as calculationparameters, which is used by the MPU 150. The RAM 154, for example,temporarily stores a program to be executed by the MPU 150.

The recording medium 156 functions as a storage unit (not shown) andstores, for example, various data such as data constituting themanipulation target image or applications. Here, an example of therecording medium 156 may include a magnetic recording medium such as ahard disk or a nonvolatile memory such as a flash memory. Further, therecording medium 156 may be detachable from the image processingapparatus 100.

The input and output interface 158 connects, for example, themanipulation input device 160 or the display device 162. Themanipulation input device 160 functions as a manipulation unit (notshown), and the display device 162 functions as a display unit (notshown). Here, examples of the input and output interface 158 may includea USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface)terminal, an HDMI (High-Definition Multimedia Interface) terminal, andvarious processing circuits. Further, the manipulation input device 160is provided, for example, on the image processing apparatus 100 andconnected to the input and output interface 158 inside the imageprocessing apparatus 100. An example of the manipulation input device160 may include buttons, direction keys, a rotary type selector such asa jog dial, or a combination thereof. Further, the display device 162 isprovided, for example, on the image processing apparatus 100 andconnected to the input and output interface 158 inside the imageprocessing apparatus 100. An example of the display device 162 mayinclude a liquid crystal display (LCD) or an organic EL display (organicElectroLuminescence display; also known as an OLED display (OrganicLight Emitting Diode display)).

Further, it is understood that the input and output interface 158 isconnected to external devices, such as a manipulation input device(e.g., a keyboard or a mouse) and a display device, that are devicesexternal to the image processing apparatus 100. Further, the displaydevice 162 may be, for example, a device in which both display and usermanipulation are possible, such as a touch screen.

The communication interface 164 is a communication medium included inthe image processing apparatus 100 and functions as the communicationunit 102 for performing wireless/wired communication with the imagingapparatus, the display device, or an external device such as a servervia a network (or directly). Here, an example of the communicationinterface 164 may include a communication antenna and an RF (RadioFrequency) circuit (wireless communication), an IEEE802.15.1 port and atransmission and reception circuit (wireless communication), anIEEE802.11b port and a transmission and reception circuit (wirelesscommunication), or a LAN (Local Area Network) terminal and atransmission and reception circuit (wired communication). Further, anexample of the network according to the present embodiment may include awired network such as a LAN or a WAN (Wide Area Network), a wirelessnetwork such as a wireless LAN (WLAN; Wireless Local Area Network) or awireless WAN (WWAN; Wireless Wide Area Network) via a base station, orthe Internet using a communication protocol, such as TCP/IP(Transmission Control Protocol/Internet Protocol).

The image processing apparatus 100 performs the process according to theimage processing method according to the present embodiment, forexample, through the configuration shown in FIG. 24. A hardwareconfiguration of the image processing apparatus 100 according to thepresent embodiment is not limited to the configuration shown in FIG. 23.For example, the image processing apparatus 100 may include an imagingdevice serving as an imaging unit (not shown) that captures a stillimage or a moving image. If the imaging device is included, the imageprocessing apparatus 100 can process, for example, the captured imagethat is generated by the imaging in the imaging device.

Here, examples of the imaging device according to the present embodimentmay include a lens/imaging element and a signal processing circuit. Thelens/imaging element includes, for example, a lens of an optical system,and an image sensor using a plurality of CMOS (Complementary Metal OxideSemiconductor) imaging elements. Further, the signal processing circuitincludes, for example, an AGC (Automatic Gain Control) circuit or an ADC(Analog to Digital Converter), and converts an analog signal generatedby the imaging element into a digital signal (image data) to performvarious signal processing. Examples of signal processing performed bythe signal processing circuit may include a white balance correctionprocess, a tint correction process, a gamma correction process, a YCbCrconversion process, and an edge enhancement process.

Further, the image processing apparatus 100 may further include, forexample, a DSP (Digital Signal Processor) and a sound output device. Anexample of the sound output device according to the present embodimentmay include an amplifier and a speaker. When the DSP and the soundoutput device are further included, the image processing apparatus 100can perform notification of, for example, gesture manipulation that ispossible in the manipulation target object, using the sound output fromthe sound output device.

Further, when the image processing apparatus 100 has, for example, aconfiguration that performs processing in a stand-alone manner, theimage processing apparatus 100 may not include the communicationinterface 164. Further, the image processing apparatus 100 may have aconfiguration in which the manipulation input device 160 or the displaydevice 162 is not included.

Referring to FIG. 23 again, an example of the configuration of the imageprocessing apparatus 100 will be described. The communication unit 102is a communication medium included in the image processing apparatus100, and performs wireless/wired communication with an external devicesuch as an imaging apparatus, a display device, or a server via thenetwork (or directly). Further, communication of the communication unit102 is controlled, for example, by the control unit 104. Here, thecommunication unit 102 may include, for example, a communication antennaand an RF circuit or a LAN terminal and a transmission and receptioncircuit, but the configuration of the communication unit 102 is notlimited thereto. For example, the communication unit 102 may have aconfiguration corresponding to any standard in which communication ispossible, such as a USB terminal and a transmission and receptioncircuit, or may have any configuration capable of communication with anexternal device via a network.

The control unit 104 includes, for example, an MPU, and serves tocontrol the entire image processing apparatus 100. Further, the controlunit 104 includes, for example, the region setting unit 110, the imageprocessing unit 112, and the display control unit 114, and plays aleading role in performing the process according to the image processingmethod according to the present embodiment.

The region setting unit 110 plays a leading role in performing theprocess (1) (region setting process), for example, to set themanipulation target region in a part of the captured image.

The image processing unit 112 plays a leading role in performing theprocess (2) (image processing) to combine the image in the displaytarget region corresponding to the manipulation target region set in theregion setting unit 110 with the manipulation target image. Further, theimage processing unit 112 may change the degree of visualization of thecaptured image, for example, based on the captured image or themanipulation target image and combine the image in the display targetregion corresponding to the manipulation target region set in the regionsetting unit 110 with the manipulation target image.

The display control unit 114 plays a leading role in the process (3)(display control process), and displays the image combined by the imageprocessing unit 112 on the display screen.

As the control unit 104 includes, for example, the region setting unit110, the image processing unit 112, and the display control unit 114,the control unit 104 plays a leading role in the processes according tothe image processing method according to the present embodiment (e.g.,the process (1) (region setting process) to the process (3) (displaycontrol process)).

Further, the configuration of the control unit according to the presentembodiment is not limited to the configuration shown in FIG. 23. Forexample, the control unit according to the present embodiment mayfurther include a notification processing unit (not shown) that performsthe process (4) (notification process). Further, the control unitaccording to the present embodiment, for example, may further include agesture recognition processing unit (not shown) that recognizes agesture in the set manipulation target region and performs a processrelated to the recognized gesture.

The image processing apparatus 100 performs the processes according tothe image processing method according to the present embodiment (e.g.,the process (1) (region setting process) to the process (3) (displaycontrol process)), for example, using the configuration shown in FIG.23. Accordingly, the image processing apparatus 100 can display thecaptured image and the manipulation target image on the display screenwhile preventing the decrease in visibility of the manipulation targetobject and preventing degradation of the maneuverability of the user,for example, using the configuration shown in FIG. 23.

Further, a configuration of the image processing apparatus according tothe present embodiment is not limited to the configuration shown in FIG.23. For example, the image processing apparatus according to the presentembodiment may individually include the region setting unit 110, theimage processing unit 112, and the display control unit 114 shown inFIG. 23, the notification processing unit (not shown), or the gesturerecognition processing unit (not shown) (realize the respective units asindividual processing circuits).

Further, the image processing apparatus according to the presentembodiment may include, for example, an imaging unit (not shown). If theimage processing apparatus includes the imaging unit (not shown), theimage processing apparatus according to the present embodiment canprocess a captured image generated by the imaging in the imaging unit(not shown).

Further, the image processing apparatus according to the presentembodiment may include, for example, a sound output unit (not shown)capable of outputting sound. Here, examples of the sound output unit(not shown) may include a DSP and a sound output device. When the imageprocessing apparatus includes the sound output unit (not shown), theimage processing apparatus according to the present embodiment canperform a variety of notifications for the user, such as notification ofa gesture manipulation possible in the manipulation target object or anerror, using the sound output by the sound output unit (not shown).

Further, when the image processing apparatus according to the presentembodiment has, for example, a configuration that performs processing ina stand-alone manner, the image processing apparatus may not include thecommunication unit 102.

As described above, the image processing apparatus according to thepresent embodiment performs, for example, the process (1) (regionsetting process), the process (2) (image processing), and the process(3) (display control process) as the processes according to the imageprocessing method according to the present embodiment. The imageprocessing apparatus according to the present embodiment sets themanipulation target region in the process (1) (region setting process).Further, the image processing apparatus according to the presentembodiment combines the image corresponding to the display target region(a region corresponding to the manipulation target region) in thecaptured image with the manipulation target image in the process (2)(image processing). Also, the combined image is displayed on the displayscreen through the process (3) (display control process).

Here, since the image processing apparatus according to the presentembodiment combines the image corresponding to the display target regioncorresponding to the set manipulation target region with themanipulation target image in the process (2) (image processing),difficulty for the user in manipulating manipulation target objectsarranged within the display screen is prevented.

Accordingly, the image processing apparatus according to the presentembodiment can display the captured image and the manipulation targetimage on the display screen while preventing degradation ofmaneuverability of the user.

Further, the image processing apparatus according to the presentembodiment may change the degree of visualization of the captured imagein the process (2) (image processing). When the degree of visualizationof the captured image is changed in the process (2) (image processing),the image processing apparatus according to the present embodiment canprevent the decrease in visibility of the manipulation target object incomparison with a case in which the captured image and the manipulationtarget image are simply combined and displayed on the display screen.Accordingly, if the degree of visualization of the captured image ischanged in the process (2) (image processing), the image processingapparatus according to the present embodiment can display the capturedimage and the manipulation target image on the display screen whilepreventing the decrease in visibility of the manipulation target objectand preventing degradation of maneuverability of the user.

Further, the image processing apparatus according to the presentembodiment sets the manipulation target region, for example, accordingto a position or a posture of the user, a size of a face or a hand, orthe like in the process (1) (region setting process). As themanipulation target region is set from the size of the face or the hand,or the like, the user can perform a gesture manipulation with the samemanipulation sense, for example, even in a position close to a camera oreven in a position away from the camera. Further, as the imageprocessing apparatus according to the present embodiment dynamicallyupdates (resets) the manipulation target region according to a movementof the detection target, such as a face or a hand, the user cancontinuously perform the manipulation even when a manipulator (user)moves during the gesture manipulation.

Further, the image processing apparatus according to the presentembodiment can display the image of the display target regioncorresponding to the manipulation target region on the display screenthrough the animation display, such as zooming, for example, through theprocess (2) (image processing) and the process (3) (display controlprocess). By displaying the image of the display target regioncorresponding to the manipulation target region on the display screenthrough the animation display, such as zooming, the image processingapparatus according to the present embodiment can cause the user to morereadily recognize a user whose gesture is currently recognized from theimage displayed on the display screen. Further, as the image of thedisplay target region corresponding to the manipulation target region isdisplayed on the display screen through the animation display, such aszooming, the manipulator (user) can recognize where the manipulationtarget region has been set, for example, even when the image processingapparatus according to the present embodiment inevitably sets themanipulation target region in a range different from a range intended bythe manipulator, such as when the manipulator protrudes outside a fieldof view of the camera. Further, as the image of the display targetregion corresponding to the manipulation target region is displayed onthe display screen through the animation display, such as zooming, theuser can recognize where the user has to move his or her hand or thelike to be able to perform a gesture manipulation even when themanipulation target region is changed according to the position andposture of the recognized user (e.g., sitting, standing, or lying down).Further, as the image of the display target region corresponding to themanipulation target region is displayed on the display screen throughthe animation display, such as zooming, the user can immediatelyrecognize that the manipulator has been switched.

The image processing apparatus has been described above as the presentembodiment, but the present embodiment is not limited to such anembodiment. The present embodiment may be applied to, for example, avariety of devices capable of processing an image and controllingdisplay on the display screen, such as a communication device such as amobile phone or a smartphone, a video/music player device (or avideo/music recording and reproducing device), a game machine, acomputer such as a PC (Personal Computer) or a server, a display devicesuch as a television receiver, or an imaging apparatus such as a digitalcamera. Further, the present embodiment may be applied to, for example,a processing IC (Integrated Circuit) that can be incorporated into suchdevices.

Further, the process according to the image processing method accordingto the present embodiment may be realized by an image processing systemincluding a plurality of devices on the assumption of a connection to anetwork (or communication between respective devices) such as, forexample, cloud computing.

Program According to the Present Embodiment

A program for causing a computer to function as the image processingapparatus according to the present embodiment (e.g., a program capableof executing the process according to the image processing methodaccording to the present embodiment, such as “the process (1) (regionsetting process) to the process (3) (display control process)” or “theprocess (1) (region setting process) to the process (4) (notificationprocess)”) is executed in the computer, thereby displaying the capturedimage and the manipulation target image on the display screen whilepreventing a decrease in the visibility of the manipulation targetobject and preventing degradation of maneuverability of the user.

Although the preferred embodiments of the present disclosure have beendescribed in detail with reference to the appended drawings, the presentdisclosure is not limited thereto. It is obvious to those skilled in theart that various modifications or variations are possible insofar asthey are within the technical scope of the appended claims or theequivalents thereof. It should be understood that such modifications orvariations are also within the technical scope of the presentdisclosure.

For example, while in the above description, provision of the program(computer program) for causing the computer to function as the imageprocessing apparatus according to the present embodiment has been shown,the present embodiment may further provide a recording medium having theprogram stored thereon together.

The configuration described above is an example of the presentembodiment and, of course, is within the technical scope of the presentdisclosure.

Additionally, the present technology may also be configured as below.

(1) An image processing apparatus including:

a region setting unit that sets a manipulation target region in a partof a captured image for feeding a gesture operation back to a user, themanipulation target region being a region in which gesture recognitionis performed;

an image processing unit that combines an image in a display targetregion with a manipulation target image to be manipulated according to aresult of the gesture recognition, the display target region being aregion corresponding to the manipulation target region to be displayedon a display screen in the captured image; and

a display control unit that displays a combined image on the displayscreen.

(2) The image processing apparatus according to (1), wherein themanipulation target region set by the region setting unit is a regionsmaller than an entire region of the captured image and larger than thedisplay target region.

(3) The image processing apparatus according to (1) or (2), wherein

the region setting unit sets the manipulation target region based on oneor two or more of a position and a size of a detection target in thecaptured image, a motion of the detection target, a posture of a user, ashape of the detection target, and a size of a manipulation targetobject included in the manipulation target image.

(4) The image processing apparatus according to any one of (1) to (3),wherein

the manipulation target image includes a region in which a gesture isrecognized and a region in which a gesture is not recognized, and

the image processing unit does not combine the captured image in theregion of the manipulation target image in which a gesture is notrecognized.

(5) The image processing apparatus according to any one of (1) to (4),wherein

the region setting unit determines whether a start condition of gesturerecognition is satisfied, and sets the manipulation target region whenthe region setting unit determines that the start condition of gesturerecognition is satisfied.

(6) The image processing apparatus according to any one of (1) to (5),wherein

when the region setting unit sets the manipulation target region, theimage processing unit sequentially changes a position and a size of aregion in the captured image to be combined with the manipulation targetimage from a position and a size of a predetermined region larger thanthe display target region to a position and a size of the display targetregion, and sequentially combines an image corresponding to a regionwhose position and size are sequentially changed with the manipulationtarget image.

(7) The image processing apparatus according to any one of (1) to (5),wherein

when the region setting unit sets a new manipulation target regiondifferent from the manipulation target region that has already been set,the image processing unit sequentially changes a position and a size ofa region in the captured image to be combined with the manipulationtarget image from a position and a size of the display target regioncorresponding to the manipulation target region that has already beenset to a position and a size of a predetermined region larger than thedisplay target region, and

when the position and the size of the region in the captured image arechanged to the position and the size of the predetermined region, theimage processing unit sequentially changes the position and the size ofthe region in the captured image from the position and the size of thepredetermined region to the position and the size of the display targetregion corresponding to the manipulation target region that has beennewly set, and

sequentially combines an image corresponding to a region whose positionand size are sequentially changed with the manipulation target image.

(8) The image processing apparatus according to any one of (1) to (5),wherein

when the region setting unit sets a new manipulation target regiondifferent from the manipulation target region that has already been set,the image processing unit changes a position of a region in the capturedimage to be combined with the manipulation target image in a manner thatthe position of the region in the captured image is moved from thedisplay target region corresponding to the manipulation target regionthat has already been set to the display target region corresponding tothe new manipulation target region that has been set.

(9) The image processing apparatus according to any one of (1) to (8),further including:

a notification processing unit that notifies a user of a gesturemanipulation possible in the manipulation target object, based on aposition of a recognition target object on which gesture recognition isperformed and a position of a manipulation target object manipulatedaccording to a result of the gesture recognition included in themanipulation target image, in the combined image.

(10) The image processing apparatus according to (1), wherein

the image processing unit changes a degree of visualization of thecaptured image based on the captured image, and combines an image in thedisplay target region in the captured image whose degree ofvisualization has been changed with the manipulation target image.

(11) The image processing apparatus according to (10), wherein

the image processing unit changes a partial range in which the capturedimage is partially visualized as a change in the degree ofvisualization.

(12) The image processing apparatus according to (11), wherein

the image processing unit determines the partial range based on apredetermined target recognized from the captured image.

(13) The image processing apparatus according to (11), wherein

the image processing unit changes a degree of transparency of thecaptured image or a display area of the captured image, as partialvisualization of the captured image.

(14) The image processing apparatus according to (10), wherein

the image processing unit changes a degree of smoothing to smooth thecaptured image as a change in the degree of the visualization.

(15) An image processing method including:

setting a manipulation target region in a part of a captured image forfeeding a gesture operation back to a user, the manipulation targetregion being a region in which gesture recognition is performed;

combining an image in a display target region with a manipulation targetimage to be manipulated according to a result of the gesturerecognition, the display target region being a region corresponding tothe manipulation target region to be displayed on a display screen inthe captured image; and

displaying a combined image on the display screen.

(16) A program for causing a computer to execute:

setting a manipulation target region in a part of a captured image forfeeding a gesture operation back to a user, the manipulation targetregion being a region in which gesture recognition is performed;

combining an image in a display target region with a manipulation targetimage to be manipulated according to a result of the gesturerecognition, the display target region being a region corresponding tothe manipulation target region to be displayed on a display screen inthe captured image; and

displaying a combined image on the display screen.

(17) An information processing system including: circuitry configuredto: acquire image data captured by an imaging device; set a manipulationtarget region based on the detection of an object in the acquired imagedata, the manipulation target region being set to have an area coveringan entire area of a display; control the display to display amanipulation target image in the area of the display corresponding tothe manipulation target region; and detect an input to the manipulationtarget image based on the acquired image data.

(18) The information processing system of (17), wherein the circuitry isconfigured to control the display to display at least a portion of theacquired image data that includes the detected object together with themanipulation target region.

(19) The information processing system of (18), wherein the circuitry isconfigured to control the display to apply a predetermined effect to theat least a portion of the acquired image data that includes the detectedobject.

(20) The information processing system of any of (18) to (19), whereinthe circuitry is configured to control the display to apply apredetermined effect to the detected object included in the at least aportion of the acquired image data.

(21) The information processing system of any of (17) to (20), whereinthe circuitry is configured to detect a hand in the acquired image data,and set the manipulation target region based on the detection of thehand.

(22) The information processing system of (21), wherein the circuitry isconfigured to set the manipulation target region based on at least oneof a size and a position of the hand detected in the image data.

(23) The information processing system of any of (21) to (22), whereinthe circuitry is configured to set, as the manipulation target region, aregion having a size including the detected hand, which is smaller thana size corresponding to the acquired image data.

(24) The information processing system of any of (17) to (23), whereinthe circuitry is configured to detect a face in the acquired image data,and set the manipulation target region based on the detection of theface.

(25) The information processing system of (24), wherein the circuitry isconfigured to set the manipulation target region based on at least oneof a size and a position of the detected face.

(26) The information processing system of any of (24) to (25), whereinthe circuitry is configured to set, as the manipulation target region, aregion having a size including the detected face, which is smaller thana size corresponding to the acquired image data.

(27) The information processing system of (24), wherein the circuitry isconfigured to: identify a user corresponding to the detected face;acquire information indicating a handedness of the identified user; andset the manipulation target region based on the handedness of the user.

(28) The information processing system of (24), wherein the circuitry isconfigured to detect a user's posture from the acquired image data, andset the manipulation target region based on the detected posture and aposition of the detected face in the acquired image data.

(29) The information processing system of any of (17) to (28), whereinthe circuitry is configured to detect at least one of a movement andshape of the object in the acquired image data, and set the manipulationtarget region based on the detected at least one of the movement andshape of the object.

(30) The information processing system of any of (17) to (29), whereinthe manipulation target image includes a plurality of selectablemanipulation target objects, and the circuitry is configured to detect,as the input to the manipulation target image, a selection of one of theplurality of selectable manipulation target objects.

(31) The information processing system of any of (17) to (30), whereinthe manipulation target image includes a first area including a firstmanipulation target object capable of being manipulated by a detectedinput, and a second area including a second manipulation target objectnot capable of being manipulated by a detected input.

(32) The information processing system of any of (17) to (31), whereinthe circuitry is configured to detect a gesture input based on theacquired image data.

(33) The information processing system of (31), wherein the circuitry isconfigured to control the display to display at least a portion of theacquired image data in the first area and not display any of the imagedata in the second area.

(34) The information processing system of (31), wherein the circuitry isconfigured to control the display to display at least a portion of theacquired image data in the first and second areas, wherein apredetermined effect is applied to the displayed at least a portion ofthe acquired image data to differentiate the at least a portion of theacquired image data displayed in the first area from the at least aportion of the acquired image data displayed in the second area.

(35) The information processing system of any of (17) to (34), whereinthe circuitry is configured to: detect a second object in the acquiredimage data; set a second manipulation target region based on thedetection of the second object from the acquired image data; and controlthe display to display notification indicating a change from themanipulation target region to the second manipulation target region.

(36) The information processing system of (35), wherein the circuitry isconfigured to control the display to display then notification byperforming a zoom-in and zoom-out on at least a portion of the acquiredimage data displayed on the display.

(37) The information processing system of any of (17) to (36), whereinthe circuitry is configured to control the display to output anotification that the detected object is outside of a range fordetecting an input to the manipulation target image based on theacquired image data.

(38) The information processing system of (37), wherein the circuitry isconfigured to control the display to output instructions indicating adirection that the detected object should move to be located in therange for detecting an input to the manipulation target image.

(39) The information processing system of any of (17) to (38), whereinthe circuitry is configured to set the manipulation target region tohave an area covering an entire area of the display.

(40) The information processing system of any of (17) to (38), whereinthe circuitry is configured to set the manipulation target region tohave a larger area than the entire area of the display.

(41) The information processing system of any of (17) to (40), whereinthe circuitry is configured to detect an input to the manipulationtarget image based on the acquired image data.

(42) A method performed by an information processing system, the methodincluding: controlling a display to display an image generated based ona manipulation target image and an image based on a manipulation targetregion that is set based on a detection of an object included inacquired image data.

(43) A non-transitory computer-readable medium includingcomputer-program instructions, which when executed by an informationprocessing system, cause the information processing system to: control adisplay to display an image generated based on a manipulation targetimage and an image based on a manipulation target region that is setbased on a detection of an object included in acquired image data.

REFERENCE SIGNS LIST

-   -   100 Image processing apparatus    -   102 Communication unit    -   104 Control unit    -   110 Region setting unit    -   112 Image processing unit    -   114 Display control unit

1. (canceled)
 2. An information processing system comprising: processingcircuitry configured to: control a display to display an image generatedbased on a manipulation target image, wherein the manipulation targetimage includes a first area including a first manipulation target objectcapable of being manipulated and a second area including a secondmanipulation target object not capable of being manipulated; and apply apredetermined effect to the displayed image to differentiate the firstarea from the second area.
 3. The system of claim 2, wherein thepredetermined effect includes display of an image that is superimposedon the first manipulation target object.
 4. The system of claim 2,wherein capable of being manipulated includes capable of having an imagesuperimposed thereupon, and not capable of being manipulated includesnot capable of having an image superimposed thereupon.
 5. The system ofclaim 3, wherein capable of being manipulated includes capable of havingan image superimposed thereupon, and and not capable of beingmanipulated includes not capable of having an image superimposedthereupon.
 6. The system of claim 2, further comprising an imagecapturing device, wherein the predetermined effect is applied based upondata received from the image capturing device.
 7. The system of claim 3,further comprising an image capturing device, wherein the predeterminedeffect is applied based upon data received from the image capturingdevice.
 8. The system of claim 2, further comprising a hand held device,wherein the predetermined effect is applied based upon data receivedfrom the hand held device.
 9. The system of claim 3, further comprisinga hand held device, wherein the predetermined effect is applied basedupon data received from the hand held device.
 10. An informationprocessing method comprising: controlling, via processing circuitry, adisplay to display an image generated based on a manipulation targetimage, wherein the manipulation target image includes a first areaincluding a first manipulation target object capable of beingmanipulated and a second area including a second manipulation targetobject not capable of being manipulated; and applying, via theprocessing circuitry, a predetermined effect to the displayed image todifferentiate the first area from the second area.
 11. The method ofclaim 10, wherein the predetermined effect includes display of an imagethat is superimposed on the first manipulation target object.
 12. Themethod of claim 10, wherein capable of being manipulated includescapable of having an image superimposed thereupon, and not capable ofbeing manipulated includes not capable of having an image superimposedthereupon.
 13. The method of claim 11, wherein capable of beingmanipulated includes capable of having an image superimposed thereupon,and not capable of being manipulated includes not capable of having animage superimposed thereupon.
 14. The method of claim 10, furthercomprising applying the predetermined effect based upon data receivedfrom an image capturing device.
 15. The method of claim 11, furthercomprising applying the predetermined effect based upon data receivedfrom an image capturing device.
 16. The method of claim 10, furthercomprising applying the predetermined effect based upon data receivedfrom a hand held device.
 17. The method of claim 11, further comprisingapplying the predetermined effect based upon data received from a handheld device.
 18. A non-transitory computer readable medium storinginstructions, which when executed by a computer, cause the computer toperform a method, the method comprising: controlling a display todisplay an image generated based on a manipulation target image, whereinthe manipulation target image includes a first area including a firstmanipulation target object capable of being manipulated and a secondarea including a second manipulation target object not capable of beingmanipulated; and applying a predetermined effect to the displayed imageto differentiate the first area from the second area.